7 ms·

Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story: Uber reinvented a ton of te

Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story: Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using. It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure. The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell. Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.

schoolornot 7y ago
- nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind - uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.
- nickv 7y ago
  Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop. I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".
- harikb 7y ago
  While I understand the point that they have money to burn, there are better ways to spend that money. Some items like “databases” are core to their SLA that it makes sense to invest. Build, source code management, etc are not “critical to de-risk”. It is not like Microsoft will one day refuse to sell them GitHub enterprise or GitHub actions because some division of Microsoft is competing with them.
- klodolph 7y ago
  That’s just apologia for NIH syndrome. It’s fine to reinvent software if that serves your goals. It’s short-sighted to reinvent software without due consideration for whether off-the-shelf solutions work. That’s not de-risking, that’s adding risk and slowing development. You see it happen at companies like that and usually it’s a sign that there’s something wrong with the culture, incentives, and the way that promotions work. My take on it is that the engineers want to make complicated solutions to hard problems to justify their salaries and get promoted, and that managers encourage that behavior so they can defend their headcount. I’m not accusing any of these people of acting in bad faith here—nobody’s reinventing tech to sabotage the company, it’s just that the system encourages this kind of behavior. This problem is not unique to Uber, you’ll see similar things happen across the industry to different degrees.
  - jeffbee 7y ago
    Well, the "due consideration" needs to duly consider the actual costs and benefits of the off-the-shelf software, not make the kind of unsupported blanket claims about its obvious superiority, like you just did. Every off-the-shelf program has maintenance costs, flaws, and weak fitness for purpose. You just made a sweeping and italicized claim that all off-the-shelf software is both less risky and faster to develop than house-made software. That's a dangerous philosophy and probably isn't true for every organization and application.
    - klodolph 7y ago
      > …not make the kind of unsupported blanket claims about its obvious superiority, like you just did… Not what I said. Let’s move on. > That's a dangerous philosophy… Go pick a fight with someone else.
decafninja 7y ago
I work at a huge non-tech company (bank), and we are guilty of the same. Tons of stuff built in-house for stupid reasons, including an entire buggy IDE that people curse every day. I have not had the privilege of working for a "tech company" where tech and tech employees are first class citizens. I'm pretty certain the in-house tooling would be much better at a company like Uber or FAANG than a company like mine.
- 18nleung 7y ago
  I interned at a FAANG last summer (Google) and I was blown away at how good the in-house tooling was (including the in-house IDE). Everything was seamless - I'd log in, tap my security key, and instantly have access to pretty much the entire monorepo and the rest of the production + deploy systems. All of the internal tools integrated with each other -- for instance, I could create a CL (i.e. a pull request) and then fix issues raised by the CI system, entirely from the IDE. "Owning the stack" completely in-house also extended to hardware -- all of my builds happened inside Google datacenters too (not on my machine), and the development box they provided me was a Chromebook.
  - klodolph 7y ago
    I’ve heard similar stories about working at Sun back in the day. It was magical, supposedly.
    - simmonmt 7y ago
      Sun didn't have a monorepo -- at least not 1998 through the acquisition. Each org of a given size (Solaris os/net, install, etc) had its own separate island of source with its own build and test systems.
      - klodolph 7y ago
        I guess my comment wasn’t clear. Talking about how you could just grab a random terminal with your smart card at Sun, and get work done. True at Google, true at Sun back in the day, not true at most places.
WoahNoun 7y ago
One of the most satisfying moves of my career was moving from a FAANG to a "cloud native" series C startup. The "not built here" syndrome is very real at the FAANG's.
- Analemma_ 7y ago
  I will defend the FAANGs just a little bit-- at least Amazon, Google and Microsoft-- in that they are now large SAAS providers and so "building it here" is essentially the same as eating your own dogfood, which is very much a good thing.
  - malandrew 7y ago
    I would also add that everyone is evaluating the tech stack of these companies in today's context. Google was founded in 1998. Facebook was founded in 2004. Uber was founded in 2009. Microsoft was founded in 1975. Amazon EC2 was first offered in 2006 and was super new and immature for a number of years. Kubernetes was first released in 2014. Mesos was a research idea for a number of years starting in 2009 and didn't reach version 1 until mid 2016. Etc. Etc. These companies generally invent their own solutions because the solutions everyone here thinks they should be using didn't exist or were not stable at the time they had a problem they needed to solve. By the time those new solutions exist and are stable enough, it requires quite a bit of investment to migrate to the newer solution as these companies already have much of their tech stack stable and bringing in revenue on their in-house solutions.
    - cbhl 7y ago
      Rebuild $infra from large company and provide it as SaaS was a pretty good recipe for a decent software business for a while. It might still be. For example, PagerDuty came out of monitoring systems at Amazon.
    - nindalf 7y ago
      I feel like what's pointed out in this comment is so obvious that it shouldn't be needed ... but evidently it is.
freewilly1040 7y ago
In the case of those larger orgs though, were the off the shelf solution was mature enough for them to use at the time?
deleted 7y ago
[deleted]
joshuamorton 7y ago
> Same thing with build systems. Bazel, Buck, Pants, Please, It's worth noting that none of these were (really) NIH. Blaze wasn't open source originally, and Buck, Pants, and Please were all essentially reimplementations of the closed source Blaze rebuilt by xooglers who went on to work at FB, Twitter, and presumably thought machine, though IDK if it's the same. Then/concurrently, Google open sourced Bazel, which is mostly-blaze.
- klodolph 7y ago
  That’s absolutely right, but I think it illustrates one of the problems with inventing your own tech stack, is that it might make perfect sense when you start the project, but several years down the road it might just be an inferior version of an open-source project. My personal sense is that this even applies to stuff like Borg, but who knows?
  - novok 7y ago
    You can't really predict the future if a large tech company will properly fund an open source project to reproduce what they have internally, and if that will be any good in time, especially if none of it exists. In the mean time, you need a solution to problem X. I've even written a few libraries like that, and still stuff out there isn't that good or does not exist years later, because it's too niche, yet still a generic issue in the industry. Can't open source it although. In an alternative timeline, Buck for example could of been the OSS solution that google decided to devote their resources to, and then in the end it would of been a good idea that FB created their own version of it.
    - klodolph 7y ago
      To be clear, what I’m saying is that “this is a problem with reinventing stuff”, I’m not saying “you shouldn’t reinvent stuff.”
  - jeffbee 7y ago
    You think Borg might be an inferior version of Kubernetes?
tmh79 7y ago
"Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available." I agree, but I'd also add that the state of the world in 2016 when most of these infra projects were launched is very different from the state of the world in 2020. Back then, uber was not easily able to run on a bunch of OSS/cloud native solutions, but now for sure its not just possible but likely the best/most responsible way to do it [source: worked there for 4+ years]
jonwalch 7y ago
Same thing at Postmates
lhorie 7y ago
I think Uber's finally recognizing this and has been adopting more off-the-shelf stuff for non-core stuff. The move to slack a few months ago comes to mind.
jmchuster 7y ago
Hard to say. I'm sure that accounts for some of it, but at a certain scale, having a system be tailor-made to fit your requirements and workflows has a lot of value. And if you tend to be towards the leading edge of scale and tech, the tools that you should be using don't exist yet or aren't known at the time without hindsight. My shop wasn't even that large, and we built our own k8s because we needed it. Looking back historically, it was even available at the time, but definitely not as the robust and popular standard that it is today. And once you have a whole system built, you're really going to have a hard time justifying re-doing your whole architecture to move to a product that does the same thing. Basically that only happens when you have a new head of engineering who comes in and goes, i don't care how many engineers you have to throw at it, and i don't care how much custom functionality you depend on that you lose, just figure how to replicate everything you do today using k8s.
caust1c 7y ago
This is revealed if you go to any of their tech talks too. That's not to discount the engineers working there, they indeed do have a number of bright engineers. But the no accountability comment is spot on.

schoolornot 7y ago

- nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind - uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.

nickv 7y ago
Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop. I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".
harikb 7y ago
While I understand the point that they have money to burn, there are better ways to spend that money. Some items like “databases” are core to their SLA that it makes sense to invest. Build, source code management, etc are not “critical to de-risk”. It is not like Microsoft will one day refuse to sell them GitHub enterprise or GitHub actions because some division of Microsoft is competing with them.
klodolph 7y ago
That’s just apologia for NIH syndrome. It’s fine to reinvent software if that serves your goals. It’s short-sighted to reinvent software without due consideration for whether off-the-shelf solutions work. That’s not de-risking, that’s adding risk and slowing development. You see it happen at companies like that and usually it’s a sign that there’s something wrong with the culture, incentives, and the way that promotions work. My take on it is that the engineers want to make complicated solutions to hard problems to justify their salaries and get promoted, and that managers encourage that behavior so they can defend their headcount. I’m not accusing any of these people of acting in bad faith here—nobody’s reinventing tech to sabotage the company, it’s just that the system encourages this kind of behavior. This problem is not unique to Uber, you’ll see similar things happen across the industry to different degrees.
- jeffbee 7y ago
  Well, the "due consideration" needs to duly consider the actual costs and benefits of the off-the-shelf software, not make the kind of unsupported blanket claims about its obvious superiority, like you just did. Every off-the-shelf program has maintenance costs, flaws, and weak fitness for purpose. You just made a sweeping and italicized claim that all off-the-shelf software is both less risky and faster to develop than house-made software. That's a dangerous philosophy and probably isn't true for every organization and application.
  - klodolph 7y ago
    > …not make the kind of unsupported blanket claims about its obvious superiority, like you just did… Not what I said. Let’s move on. > That's a dangerous philosophy… Go pick a fight with someone else.

decafninja 7y ago

I work at a huge non-tech company (bank), and we are guilty of the same. Tons of stuff built in-house for stupid reasons, including an entire buggy IDE that people curse every day. I have not had the privilege of working for a "tech company" where tech and tech employees are first class citizens. I'm pretty certain the in-house tooling would be much better at a company like Uber or FAANG than a company like mine.

18nleung 7y ago
I interned at a FAANG last summer (Google) and I was blown away at how good the in-house tooling was (including the in-house IDE). Everything was seamless - I'd log in, tap my security key, and instantly have access to pretty much the entire monorepo and the rest of the production + deploy systems. All of the internal tools integrated with each other -- for instance, I could create a CL (i.e. a pull request) and then fix issues raised by the CI system, entirely from the IDE. "Owning the stack" completely in-house also extended to hardware -- all of my builds happened inside Google datacenters too (not on my machine), and the development box they provided me was a Chromebook.
- klodolph 7y ago
  I’ve heard similar stories about working at Sun back in the day. It was magical, supposedly.
  - simmonmt 7y ago
    Sun didn't have a monorepo -- at least not 1998 through the acquisition. Each org of a given size (Solaris os/net, install, etc) had its own separate island of source with its own build and test systems.
    - klodolph 7y ago
      I guess my comment wasn’t clear. Talking about how you could just grab a random terminal with your smart card at Sun, and get work done. True at Google, true at Sun back in the day, not true at most places.

WoahNoun 7y ago

One of the most satisfying moves of my career was moving from a FAANG to a "cloud native" series C startup. The "not built here" syndrome is very real at the FAANG's.

Analemma_ 7y ago
I will defend the FAANGs just a little bit-- at least Amazon, Google and Microsoft-- in that they are now large SAAS providers and so "building it here" is essentially the same as eating your own dogfood, which is very much a good thing.
- malandrew 7y ago
  I would also add that everyone is evaluating the tech stack of these companies in today's context. Google was founded in 1998. Facebook was founded in 2004. Uber was founded in 2009. Microsoft was founded in 1975. Amazon EC2 was first offered in 2006 and was super new and immature for a number of years. Kubernetes was first released in 2014. Mesos was a research idea for a number of years starting in 2009 and didn't reach version 1 until mid 2016. Etc. Etc. These companies generally invent their own solutions because the solutions everyone here thinks they should be using didn't exist or were not stable at the time they had a problem they needed to solve. By the time those new solutions exist and are stable enough, it requires quite a bit of investment to migrate to the newer solution as these companies already have much of their tech stack stable and bringing in revenue on their in-house solutions.
  - cbhl 7y ago
    Rebuild $infra from large company and provide it as SaaS was a pretty good recipe for a decent software business for a while. It might still be. For example, PagerDuty came out of monitoring systems at Amazon.
  - nindalf 7y ago
    I feel like what's pointed out in this comment is so obvious that it shouldn't be needed ... but evidently it is.

freewilly1040 7y ago

In the case of those larger orgs though, were the off the shelf solution was mature enough for them to use at the time?

deleted 7y ago

[deleted]

joshuamorton 7y ago

> Same thing with build systems. Bazel, Buck, Pants, Please, It's worth noting that none of these were (really) NIH. Blaze wasn't open source originally, and Buck, Pants, and Please were all essentially reimplementations of the closed source Blaze rebuilt by xooglers who went on to work at FB, Twitter, and presumably thought machine, though IDK if it's the same. Then/concurrently, Google open sourced Bazel, which is mostly-blaze.

klodolph 7y ago
That’s absolutely right, but I think it illustrates one of the problems with inventing your own tech stack, is that it might make perfect sense when you start the project, but several years down the road it might just be an inferior version of an open-source project. My personal sense is that this even applies to stuff like Borg, but who knows?
- novok 7y ago
  You can't really predict the future if a large tech company will properly fund an open source project to reproduce what they have internally, and if that will be any good in time, especially if none of it exists. In the mean time, you need a solution to problem X. I've even written a few libraries like that, and still stuff out there isn't that good or does not exist years later, because it's too niche, yet still a generic issue in the industry. Can't open source it although. In an alternative timeline, Buck for example could of been the OSS solution that google decided to devote their resources to, and then in the end it would of been a good idea that FB created their own version of it.
  - klodolph 7y ago
    To be clear, what I’m saying is that “this is a problem with reinventing stuff”, I’m not saying “you shouldn’t reinvent stuff.”
- jeffbee 7y ago
  You think Borg might be an inferior version of Kubernetes?

tmh79 7y ago

"Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available." I agree, but I'd also add that the state of the world in 2016 when most of these infra projects were launched is very different from the state of the world in 2020. Back then, uber was not easily able to run on a bunch of OSS/cloud native solutions, but now for sure its not just possible but likely the best/most responsible way to do it [source: worked there for 4+ years]

jonwalch 7y ago

Same thing at Postmates

lhorie 7y ago

I think Uber's finally recognizing this and has been adopting more off-the-shelf stuff for non-core stuff. The move to slack a few months ago comes to mind.

jmchuster 7y ago

Hard to say. I'm sure that accounts for some of it, but at a certain scale, having a system be tailor-made to fit your requirements and workflows has a lot of value. And if you tend to be towards the leading edge of scale and tech, the tools that you should be using don't exist yet or aren't known at the time without hindsight. My shop wasn't even that large, and we built our own k8s because we needed it. Looking back historically, it was even available at the time, but definitely not as the robust and popular standard that it is today. And once you have a whole system built, you're really going to have a hard time justifying re-doing your whole architecture to move to a product that does the same thing. Basically that only happens when you have a new head of engineering who comes in and goes, i don't care how many engineers you have to throw at it, and i don't care how much custom functionality you depend on that you lose, just figure how to replicate everything you do today using k8s.

caust1c 7y ago

This is revealed if you go to any of their tech talks too. That's not to discount the engineers working there, they indeed do have a number of bright engineers. But the no accountability comment is spot on.