10 ms·

This is hardly surprising at all. There used to be lots of complaints, both internally and publicly, about how bloated and ineffective Uber's engineering organi

This is hardly surprising at all. There used to be lots of complaints, both internally and publicly, about how bloated and ineffective Uber's engineering organization was. Many engineers were bitter about many overlapping projects in Uber. It's also funny that Uber engineers used many bogus reasons to implement NIH projects. A typical example is that Uber had a team to implement their own Bazel-like system because "Bazel (or any other tool) does not scale in Uber-scale". Same went for their own GPU database, their own SQS, their own key-value store, their own resource scheduler, their own datacenter, and their own deployment system. The end result was that almost everything sucked. Engineers suffered productivity loss for years. I kid you not that Uber's docker system didn't support persistent volume for more than three years, so anyone who wanted to build a persistent system like Elasticsearch had to jump hoops and spent weeks even just to get machines. And the sad fact is, Uber did not have a "Uber scale". The traffic and complexity it had to handle was far less than other internet companies. I wish they didn't hire so irresponsibly back in 2015 and 2016.

fishywang 7y ago
I can understand Uber wanting to build other tools in house because the existing open source ones might not scale in Uber-scale. But Bazel? That's just absurd.
lhorie 7y ago
> A typical example is that Uber had a team to implement their own Bazel-like system because "Bazel (or any other tool) does not scale in Uber-scale" Citation needed? AFAIK Uber was using Facebook's Buck and decided to move to Bazel for stacks where Buck is admittedly not good at (e.g. Go). I'm not aware of any org within Uber reinventing Bazel. The web org might be closest to what you're claiming, but even there, it merely wraps over Bazel to bridge over its shortcomings (i.e. package management is typically done "outside" of Bazel, via yarn/npm in web, as seen in the official rules_nodejs and other 3rd party spin-off rulesets)
- mooted1 7y ago
  This was under consideration at one time by the Seattle build team and was a recurring point of discussion during infra summits. Unclear if we actually broke ground.
  - hintymad 7y ago
    And the team publicly defended their consideration in a company all-hands, IIRC.
  - lhorie 7y ago
    Just saw your other comment. Not sure exactly when you left in 2018, but around that time, there was an internal summit involving various platform/infra orgs that decided to invest in Bazel as a company-wide standard. Java is still on Buck due to some technical reasons, but several orgs are either moving towards or already fully on Bazel.
mooted1 7y ago
I worked at Uber from 2014-2018. Can confirm this. It was a very poorly managed org because it was extremely grassroots driven—leadership was underempowered to say no to front line teams or hold groups accountable. The overlaps and poorly considered projects described above are not an exaggeration. Many of them were designed for building the lead engineer's open source brand, not for company needs. Half of the projects described above have since been canceled. Google: - Hyperbahn - Cherami - Schemaless - Peloton - uDeploy - m3db - piper - XYS Contrary to their glossy open source and tech blog presentations, these projects were highly contentious internally and widely viewed as inferior to industry counterparts. Each of these projects had 5-20 engineers working on them for over a year; many of them are being EOLed or phased out currently. And this is just the list that I can publicly talk about. This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.
- devicetray0 7y ago
  > I worked at Uber from 2014-2018. Seems like you probably got a lot of shares and were lucky. Now that they IPO'd, do you consider that you came out ahead financially (compared to working elsewhere)?
  - tempsy 7y ago
    Just a note: Uber's valuation was higher in 2015 than it is right now...
    - usaar333 7y ago
      It was $40b at start of 2015 and was rumored to be at $50b mid 2015. It's $53b currently. That said with dilution, it's likely 2015 share price is less than it is today.
    - mlinsey 7y ago
      No inside info on Uber, but it is common for options at private companies to have strike prices considerably below reported valuations, which is proper (and adjudicated by a third party) considering the reported private valuations usually based on preferred shares that come with extra rights. That said, I dunno what year Uber switched to RSU’s, which would make this moot.
iamasoftwaredev 7y ago
> their own key-value store It seems like everyone does that. and it makes sense cause if given any chance to do it I'd jump on it immediately.
- hintymad 7y ago
  Not in 2015/2016 when so many choices were already available, and certainly not in an org where MySQL would just be more than enough to meet Uber's need. Besides, Uber built their version on top of MySQL following pretty much FriendFeed's design: https://backchannel.org/blog/friendfeed-schemaless-mysql https://backchannel.org/blog/friendfeed-schemaless-mysql. A strange choice by itself given that better solutions had popped up left and right during the 6 years since the blog was published. Uber switch to Schemaless from a Postgres cluster, claiming that Postgres "does not scale for Uber". I see irony in that piece of Uber history. Particularly, Uber always claimed that some OS solution "had serious reliability issues". Cassandra in 2014 worked for companies that had orders of magnitude of traffic, yet Uber found it unusable. Similarly, Uber switched from MySQL to Postgres and then from Postgres to MySQL. Each time Uber published an article to claim engineering victory. The truth is that any of the three systems would be perfectly fine for Uber. Oh, didn't someone also claim that Elaticsearch crashed under Uber's load" and JVM-based systems were not usable for Uber's scale? " before Uber ended up adopting Elasticsearch? What really happened was that some teams in Uber simply didn't want to dive deep and produce objective tech assessment.
  - iamasoftwaredev 7y ago
    I meant it makes sense that people keep building them needlessly because I would just at the opportunity even if I felt like it was totally pointless.
  - deleted 7y ago
    [deleted]
  - deleted 7y ago
    [deleted]
- hinkley 7y ago
xbeta 7y ago
The lays off this wave is mostly operations, marketing, and other areas. Not in core engineering.
mytailorisrich 7y ago
It's a self-sustaining reaction in any organisation left unchecked: Manager has an incentive to hire below him, create projects to justify jobs, hire more, create more projects, etc.
- nilkn 7y ago
  There needs to be a way to reward managers for running teams/organizations that have a big impact with low headcount. After a certain point, in most organizations, the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly.
  - mytailorisrich 7y ago
    > the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly. Yes, that's exactly how it works. It helps you move from manager to senior manager. It helps your boss move from senior manager to director, etc. Growing your team is management 101 for anyone with a minimum politically astute. So if the message you get "from above" is that the budget is unlimited (which honestly seems to have been the case at Uber) then you dream up any justification to increase your headcount.
    - coldcode 7y ago
      At my (large but not tech) employer, some mgr/execs hire reams of contractors to pad the headcount, then have to beg/borrow/steal budget to keep them around until they get promoted.
      - 9nGQluzmnq3M 7y ago
        Contractors typically come from a different budget than headcount. So when you can't hire employees, you hire contractors, and then use them to justify your next headcount request. Rinse and repeat.
wutbrodo 7y ago
> Bazel (or any other tool) does not scale in Uber-scale This is hilarious, given they Bazel is the public version of Google-internal Blaze
- hef19898 7y ago
  Not Uber related, but as soon as ERP systems or warehouse management or logistics and supply chain management are concerned I sometimes have the impression that a lot of "start-ups" are falling in the same trap. There is simply no way to beat the likes of SAP, Microsoft or even Oracle in the sheer man hours, resources and research spent on their systems. And yes, they are cumbersome and UX sicks. But then they are extremely powerful out of the box, and completely auditable. Maybe it was necessary back when Amazon started due to limitations with webshop integration (or existaznce as far that is concerned). Nowadays not so much.
  - 1MoreThing 7y ago
    > completely auditable The biggest thing that people who try to roll their own fail on is this right here.
    - rkwz 7y ago
      Curious, can you elaborate on what “completely auditable” means?
      - yellowapple 7y ago
        In short: the system should log every operation performed within, and that log should be readily accessible to (authorized) end-users. For example: let's say you have a warehouse management system, which tracks inventory movements in a warehouse / fulfillment center that ships, say, toothbrushes to customers. Multiple customers call in with complaints about getting blue toothbrushes when they ordered red toothbrushes. Seeing as how this is clearly not an isolated incident (multiple people are upset about not getting the right toothbrush!), it's up to you to find out where in the whole lifecycle of that toothbrush in your warehouse - all the way from when it was received to when it was shipped - that error might be happening. If your warehouse management system is completely auditable, then you'll be able query the transaction log (given perhaps the affected order and item numbers) to work backwards to investigate where the fault occurred, be it in shipping, picking, replenishment, receiving, the vendor, or what have you. The "incumbent" off-the-shelf ERP and WMS vendors tend to support this sort of comprehensive audit trail much more thoroughly than those writing new ones (whether as an in-house project or as a startup attempting to disrupt the space), simply because those incumbents have already had their hard-earned lessons on why literally every transaction must be thoroughly logged (whereas the newcomers and DIYers typically have not, or are in the process of learning as they encounter more and more situations where that audit trail is necessary).
hinkley 7y ago
A couple news stories back, it came out how much money Uber was spending on Eats and it was simply staggering. My company has similar stories. When you look at the amount of requests per second we're actually dealing with, there are plenty of other companies doing something far less "sophisticated" and handling more traffic. Keep it Simple, Stupid. Allowing for heterogeneity is a blindspot for a lot of wannabe architects. We have a system where most of the code is dominated by 2 platforms and there is no way to democratically add/test/demo new routes that don't go through one of those two. And every change has to be able to deal with production traffic levels from word one. Similar with deployment management. In this sort of climate, there is no incentive to ever go back and revisit NIH vs off-the-shelf tools that has now outstripped the functionality of the in-house tool. When the friction is so high it's a non-starter. And when some of those NIH projects started in ignorance (ie, a tool existed and was already mature enough to start using, but the author didn't find it) it represents extra frustration. At some point your engineers figure out nothing they are working on will present well on their resume, and they either get burnt out or flea.
- hedvig 7y ago
  >At some point your engineers figure out nothing they are >working on will present well on their resume, and they >either get burnt out or flea. Shouldn't the fact that they were part of standing up solid reasonable solutions reflect well on their resume, to reasonable hiring managers elsewhere?
kalessin 7y ago
I agree with how engineering was disorganized at Uber but it makes sense to me to not trust your competitors with your infrastructure. Some projects worked well, Schemaless is still going strong and I'd take it any day over DynamoDB or Cassandra.
yashap 7y ago
> Uber engineers used many bogus reasons to implement NIH projects In my experience this is one of the most toxic diseases an engineering org can catch. It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. They come up with a BS reason to re-implement some rock solid 3rd party tech, spend ages doing it, only to leave the company as it starts to comes crumbling down. At that point other engineers have to put in massive effort to rip out their buggy timesink, and replace it with the open source library/database/protocol/whatever that was the obvious choice from the start. If people like this exist in your org, you either need to convince them of the error of their ways, or let them go. The damage they cause when left unchecked is just too great.
- john_moscow 7y ago
  >It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. Well, if the company's business is connecting drivers with riders, there isn't much engineering involved in growing the company. Pricing, handling investors, PR/advertising and working with municipal governments does affect your growth. The exact details on how you store the destination's GPS coordinates in the database don't really matter. A good product manager who understands usability and can communicate their ideas to the management can add some value, but having a 20% lower price than the competitors do will make 90% of the users ignore any usability hiccups. Hiring thousands of engineers to maintain a process that only needs a handful is perfectly reasonable, since the investors don't understand tech and generally like the head count. So this is critical for raising VC capital needed to subsidize the rides, that in turns gives great growth numbers. A typical developer doesn't understand these mechanics. And even if they did, the only alternative to delivering useless unneeded results would be to resign, except good luck finding a competitively paying non-bullshit job in the current market.
  - golemiprague 7y ago
    That's not true, companies like instagram for example didn't have many developers when they got sold for billions. The investors don't want their money wasted, they want the growth even when not profitable but they are not stupid and if a company can achieve this growth with few developers they will be more than happy to support it.
hw 7y ago
two words - job security I've been in many orgs where engineers like to reinvent the wheel. Just so they can claim ownership over more components, add polish to resumes ("Led project to rewrite X to scale", "built X from scratch") and makes them look busy. When it comes to perf reviews, or when it comes to job cuts, they'd likely survive given they know more / built those components.
_bxg1 7y ago
What are NIH projects?
- johns 7y ago
  Not invented here
- deleted 7y ago
  [deleted]
- Infinity15 7y ago
  Not-Invented-Here
- runeb 7y ago
  "Not invented here"
- grogenaut 7y ago
  national institute of health, aka healthcare
mumblemumble 7y ago
FWIW, I used to work at a company with a severe case of NIH syndrome, but it worked out well. From what I could tell, the difference was all about mentality: This company did it, not because of weak management, but because of strong management that was trying to push a culture of KISS and technical ownership. So, whenever a technical team identified a need for X, there was a lot of time putting into understanding the problem, understanding the company's real needs, and figuring out the simplest thing that would get the job done. If there was an off-the-shelf product that did one thing and did it well (e.g., protocol buffers for serializing datagrams), that would get chosen. If everything out there was a confusing pile of feature creep that was trying to be all things to all people (e.g., configuration management), an in-house product would get launched. TBH, the tech stack was a joy to work on. Especially that home-grown configuration management system. So simple, so effective. I've never worked for Uber or anything, but I've attended a few lectures and conference sessions where Uber engineers presented things they worked on, and my sense was that that wasn't at all what was happening there. Most the projects I saw seemed to be motivated by resume-driven development. Which is absolutely the way to go if you want to end up with solutions that are complicated, hard to manage, and half-implemented.
- freedomben 7y ago
  Are any of these projects open source, and if so can you point me toward some of them?
  - mumblemumble 7y ago
    Nope. They had no particular business interest in doing so, and, from a technical perspective, doing so would have created a conflict between the needs of the community and the needs of the company. I'm not sure a lot of it would have flown, anyway, due to going against the grain of modern software development. For example, automated package management was banned at the company level. Instead there was a central repository of approved packages that everyone used. That's actually another great example of figuring out the simplest solution that would work: It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization. It gave tech ops a really, really easy place to go look when they needed to do security audits on our dependencies or for pushing security updates. And it made it easy for legal to keep an eye on compliance. Stuff like that. Honestly, I thought it worked really, really, amazingly well. And I'd have never thought of it, because it flies in the face of 20 years' worth of the entire software field pulling things in in a completely different, much fancier and more complicated and bell-y and whistle-y, direction. Also, most of what was going on was so simple you wouldn't even be able to turn it into an open source project. Like, I know that the official wisdom is that you're not supposed to route your event streaming system through a database, but wow, assuming you're willing to shell out for a commercial RDBMS, it's amazing how far you can scale an event streaming mechanism that routes everything through database tables. And don't even get me started on how easy it was to support in production. Sometimes I have to remind myself, open source is only free when your programmers aren't drawing a paycheck.
deleted 7y ago
[deleted]
sharadov 7y ago
What's NIH?
- callalex 7y ago
  https://en.m.wikipedia.org/wiki/Not_invented_here https://en.m.wikipedia.org/wiki/Not_invented_here
- deleted 7y ago
  [deleted]
twic 7y ago
No, no, Uber employed all those developers because every one of them's work was crucial to the success of the company! Dan Luu wrote a famous essay about it, and he can hardly be wrong, can he?!? https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/
sssdmmmm 7y ago
What I would love to know is how many people built these homebrewed systems at Google in 2007, then jumped to Facebook and built them again in 2010, and on to Uber to do it yet again in 2013.

fishywang 7y ago

I can understand Uber wanting to build other tools in house because the existing open source ones might not scale in Uber-scale. But Bazel? That's just absurd.

lhorie 7y ago

> A typical example is that Uber had a team to implement their own Bazel-like system because "Bazel (or any other tool) does not scale in Uber-scale" Citation needed? AFAIK Uber was using Facebook's Buck and decided to move to Bazel for stacks where Buck is admittedly not good at (e.g. Go). I'm not aware of any org within Uber reinventing Bazel. The web org might be closest to what you're claiming, but even there, it merely wraps over Bazel to bridge over its shortcomings (i.e. package management is typically done "outside" of Bazel, via yarn/npm in web, as seen in the official rules_nodejs and other 3rd party spin-off rulesets)

mooted1 7y ago
This was under consideration at one time by the Seattle build team and was a recurring point of discussion during infra summits. Unclear if we actually broke ground.
- hintymad 7y ago
  And the team publicly defended their consideration in a company all-hands, IIRC.
- lhorie 7y ago
  Just saw your other comment. Not sure exactly when you left in 2018, but around that time, there was an internal summit involving various platform/infra orgs that decided to invest in Bazel as a company-wide standard. Java is still on Buck due to some technical reasons, but several orgs are either moving towards or already fully on Bazel.

mooted1 7y ago

I worked at Uber from 2014-2018. Can confirm this. It was a very poorly managed org because it was extremely grassroots driven—leadership was underempowered to say no to front line teams or hold groups accountable. The overlaps and poorly considered projects described above are not an exaggeration. Many of them were designed for building the lead engineer's open source brand, not for company needs. Half of the projects described above have since been canceled. Google: - Hyperbahn - Cherami - Schemaless - Peloton - uDeploy - m3db - piper - XYS Contrary to their glossy open source and tech blog presentations, these projects were highly contentious internally and widely viewed as inferior to industry counterparts. Each of these projects had 5-20 engineers working on them for over a year; many of them are being EOLed or phased out currently. And this is just the list that I can publicly talk about. This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.

devicetray0 7y ago
> I worked at Uber from 2014-2018. Seems like you probably got a lot of shares and were lucky. Now that they IPO'd, do you consider that you came out ahead financially (compared to working elsewhere)?
- tempsy 7y ago
  Just a note: Uber's valuation was higher in 2015 than it is right now...
  - usaar333 7y ago
    It was $40b at start of 2015 and was rumored to be at $50b mid 2015. It's $53b currently. That said with dilution, it's likely 2015 share price is less than it is today.
  - mlinsey 7y ago
    No inside info on Uber, but it is common for options at private companies to have strike prices considerably below reported valuations, which is proper (and adjudicated by a third party) considering the reported private valuations usually based on preferred shares that come with extra rights. That said, I dunno what year Uber switched to RSU’s, which would make this moot.

iamasoftwaredev 7y ago

> their own key-value store It seems like everyone does that. and it makes sense cause if given any chance to do it I'd jump on it immediately.

hintymad 7y ago
Not in 2015/2016 when so many choices were already available, and certainly not in an org where MySQL would just be more than enough to meet Uber's need. Besides, Uber built their version on top of MySQL following pretty much FriendFeed's design: https://backchannel.org/blog/friendfeed-schemaless-mysql https://backchannel.org/blog/friendfeed-schemaless-mysql. A strange choice by itself given that better solutions had popped up left and right during the 6 years since the blog was published. Uber switch to Schemaless from a Postgres cluster, claiming that Postgres "does not scale for Uber". I see irony in that piece of Uber history. Particularly, Uber always claimed that some OS solution "had serious reliability issues". Cassandra in 2014 worked for companies that had orders of magnitude of traffic, yet Uber found it unusable. Similarly, Uber switched from MySQL to Postgres and then from Postgres to MySQL. Each time Uber published an article to claim engineering victory. The truth is that any of the three systems would be perfectly fine for Uber. Oh, didn't someone also claim that Elaticsearch crashed under Uber's load" and JVM-based systems were not usable for Uber's scale? " before Uber ended up adopting Elasticsearch? What really happened was that some teams in Uber simply didn't want to dive deep and produce objective tech assessment.
- iamasoftwaredev 7y ago
  I meant it makes sense that people keep building them needlessly because I would just at the opportunity even if I felt like it was totally pointless.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
hinkley 7y ago

xbeta 7y ago

The lays off this wave is mostly operations, marketing, and other areas. Not in core engineering.

mytailorisrich 7y ago

It's a self-sustaining reaction in any organisation left unchecked: Manager has an incentive to hire below him, create projects to justify jobs, hire more, create more projects, etc.

nilkn 7y ago
There needs to be a way to reward managers for running teams/organizations that have a big impact with low headcount. After a certain point, in most organizations, the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly.
- mytailorisrich 7y ago
  > the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly. Yes, that's exactly how it works. It helps you move from manager to senior manager. It helps your boss move from senior manager to director, etc. Growing your team is management 101 for anyone with a minimum politically astute. So if the message you get "from above" is that the budget is unlimited (which honestly seems to have been the case at Uber) then you dream up any justification to increase your headcount.
  - coldcode 7y ago
    At my (large but not tech) employer, some mgr/execs hire reams of contractors to pad the headcount, then have to beg/borrow/steal budget to keep them around until they get promoted.
    - 9nGQluzmnq3M 7y ago
      Contractors typically come from a different budget than headcount. So when you can't hire employees, you hire contractors, and then use them to justify your next headcount request. Rinse and repeat.

wutbrodo 7y ago

> Bazel (or any other tool) does not scale in Uber-scale This is hilarious, given they Bazel is the public version of Google-internal Blaze

hef19898 7y ago
Not Uber related, but as soon as ERP systems or warehouse management or logistics and supply chain management are concerned I sometimes have the impression that a lot of "start-ups" are falling in the same trap. There is simply no way to beat the likes of SAP, Microsoft or even Oracle in the sheer man hours, resources and research spent on their systems. And yes, they are cumbersome and UX sicks. But then they are extremely powerful out of the box, and completely auditable. Maybe it was necessary back when Amazon started due to limitations with webshop integration (or existaznce as far that is concerned). Nowadays not so much.
- 1MoreThing 7y ago
  > completely auditable The biggest thing that people who try to roll their own fail on is this right here.
  - rkwz 7y ago
    Curious, can you elaborate on what “completely auditable” means?
    - yellowapple 7y ago
      In short: the system should log every operation performed within, and that log should be readily accessible to (authorized) end-users. For example: let's say you have a warehouse management system, which tracks inventory movements in a warehouse / fulfillment center that ships, say, toothbrushes to customers. Multiple customers call in with complaints about getting blue toothbrushes when they ordered red toothbrushes. Seeing as how this is clearly not an isolated incident (multiple people are upset about not getting the right toothbrush!), it's up to you to find out where in the whole lifecycle of that toothbrush in your warehouse - all the way from when it was received to when it was shipped - that error might be happening. If your warehouse management system is completely auditable, then you'll be able query the transaction log (given perhaps the affected order and item numbers) to work backwards to investigate where the fault occurred, be it in shipping, picking, replenishment, receiving, the vendor, or what have you. The "incumbent" off-the-shelf ERP and WMS vendors tend to support this sort of comprehensive audit trail much more thoroughly than those writing new ones (whether as an in-house project or as a startup attempting to disrupt the space), simply because those incumbents have already had their hard-earned lessons on why literally every transaction must be thoroughly logged (whereas the newcomers and DIYers typically have not, or are in the process of learning as they encounter more and more situations where that audit trail is necessary).

hinkley 7y ago

A couple news stories back, it came out how much money Uber was spending on Eats and it was simply staggering. My company has similar stories. When you look at the amount of requests per second we're actually dealing with, there are plenty of other companies doing something far less "sophisticated" and handling more traffic. Keep it Simple, Stupid. Allowing for heterogeneity is a blindspot for a lot of wannabe architects. We have a system where most of the code is dominated by 2 platforms and there is no way to democratically add/test/demo new routes that don't go through one of those two. And every change has to be able to deal with production traffic levels from word one. Similar with deployment management. In this sort of climate, there is no incentive to ever go back and revisit NIH vs off-the-shelf tools that has now outstripped the functionality of the in-house tool. When the friction is so high it's a non-starter. And when some of those NIH projects started in ignorance (ie, a tool existed and was already mature enough to start using, but the author didn't find it) it represents extra frustration. At some point your engineers figure out nothing they are working on will present well on their resume, and they either get burnt out or flea.

hedvig 7y ago
>At some point your engineers figure out nothing they are >working on will present well on their resume, and they >either get burnt out or flea. Shouldn't the fact that they were part of standing up solid reasonable solutions reflect well on their resume, to reasonable hiring managers elsewhere?

kalessin 7y ago

I agree with how engineering was disorganized at Uber but it makes sense to me to not trust your competitors with your infrastructure. Some projects worked well, Schemaless is still going strong and I'd take it any day over DynamoDB or Cassandra.

yashap 7y ago

> Uber engineers used many bogus reasons to implement NIH projects In my experience this is one of the most toxic diseases an engineering org can catch. It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. They come up with a BS reason to re-implement some rock solid 3rd party tech, spend ages doing it, only to leave the company as it starts to comes crumbling down. At that point other engineers have to put in massive effort to rip out their buggy timesink, and replace it with the open source library/database/protocol/whatever that was the obvious choice from the start. If people like this exist in your org, you either need to convince them of the error of their ways, or let them go. The damage they cause when left unchecked is just too great.

john_moscow 7y ago
>It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. Well, if the company's business is connecting drivers with riders, there isn't much engineering involved in growing the company. Pricing, handling investors, PR/advertising and working with municipal governments does affect your growth. The exact details on how you store the destination's GPS coordinates in the database don't really matter. A good product manager who understands usability and can communicate their ideas to the management can add some value, but having a 20% lower price than the competitors do will make 90% of the users ignore any usability hiccups. Hiring thousands of engineers to maintain a process that only needs a handful is perfectly reasonable, since the investors don't understand tech and generally like the head count. So this is critical for raising VC capital needed to subsidize the rides, that in turns gives great growth numbers. A typical developer doesn't understand these mechanics. And even if they did, the only alternative to delivering useless unneeded results would be to resign, except good luck finding a competitively paying non-bullshit job in the current market.
- golemiprague 7y ago
  That's not true, companies like instagram for example didn't have many developers when they got sold for billions. The investors don't want their money wasted, they want the growth even when not profitable but they are not stupid and if a company can achieve this growth with few developers they will be more than happy to support it.

hw 7y ago

two words - job security I've been in many orgs where engineers like to reinvent the wheel. Just so they can claim ownership over more components, add polish to resumes ("Led project to rewrite X to scale", "built X from scratch") and makes them look busy. When it comes to perf reviews, or when it comes to job cuts, they'd likely survive given they know more / built those components.

_bxg1 7y ago

What are NIH projects?

johns 7y ago
Not invented here
deleted 7y ago
[deleted]
Infinity15 7y ago
Not-Invented-Here
runeb 7y ago
"Not invented here"
grogenaut 7y ago
national institute of health, aka healthcare

mumblemumble 7y ago

FWIW, I used to work at a company with a severe case of NIH syndrome, but it worked out well. From what I could tell, the difference was all about mentality: This company did it, not because of weak management, but because of strong management that was trying to push a culture of KISS and technical ownership. So, whenever a technical team identified a need for X, there was a lot of time putting into understanding the problem, understanding the company's real needs, and figuring out the simplest thing that would get the job done. If there was an off-the-shelf product that did one thing and did it well (e.g., protocol buffers for serializing datagrams), that would get chosen. If everything out there was a confusing pile of feature creep that was trying to be all things to all people (e.g., configuration management), an in-house product would get launched. TBH, the tech stack was a joy to work on. Especially that home-grown configuration management system. So simple, so effective. I've never worked for Uber or anything, but I've attended a few lectures and conference sessions where Uber engineers presented things they worked on, and my sense was that that wasn't at all what was happening there. Most the projects I saw seemed to be motivated by resume-driven development. Which is absolutely the way to go if you want to end up with solutions that are complicated, hard to manage, and half-implemented.

freedomben 7y ago
Are any of these projects open source, and if so can you point me toward some of them?
- mumblemumble 7y ago
  Nope. They had no particular business interest in doing so, and, from a technical perspective, doing so would have created a conflict between the needs of the community and the needs of the company. I'm not sure a lot of it would have flown, anyway, due to going against the grain of modern software development. For example, automated package management was banned at the company level. Instead there was a central repository of approved packages that everyone used. That's actually another great example of figuring out the simplest solution that would work: It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization. It gave tech ops a really, really easy place to go look when they needed to do security audits on our dependencies or for pushing security updates. And it made it easy for legal to keep an eye on compliance. Stuff like that. Honestly, I thought it worked really, really, amazingly well. And I'd have never thought of it, because it flies in the face of 20 years' worth of the entire software field pulling things in in a completely different, much fancier and more complicated and bell-y and whistle-y, direction. Also, most of what was going on was so simple you wouldn't even be able to turn it into an open source project. Like, I know that the official wisdom is that you're not supposed to route your event streaming system through a database, but wow, assuming you're willing to shell out for a commercial RDBMS, it's amazing how far you can scale an event streaming mechanism that routes everything through database tables. And don't even get me started on how easy it was to support in production. Sometimes I have to remind myself, open source is only free when your programmers aren't drawing a paycheck.

deleted 7y ago

[deleted]

sharadov 7y ago

What's NIH?

callalex 7y ago
https://en.m.wikipedia.org/wiki/Not_invented_here https://en.m.wikipedia.org/wiki/Not_invented_here
deleted 7y ago
[deleted]

twic 7y ago

No, no, Uber employed all those developers because every one of them's work was crucial to the success of the company! Dan Luu wrote a famous essay about it, and he can hardly be wrong, can he?!? https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/

sssdmmmm 7y ago

What I would love to know is how many people built these homebrewed systems at Google in 2007, then jumped to Facebook and built them again in 2010, and on to Uber to do it yet again in 2013.