6 ms·

When I worked at Uber, this project was openly mocked. One of the CTO’s biggest failures was implementing a promotion scheme where you needed to create a new se

When I worked at Uber, this project was openly mocked. One of the CTO’s biggest failures was implementing a promotion scheme where you needed to create a new service in order to be considered “innovative”. This promotion scheme marked what I consider the end of Uber’s engineering excellence and the start of what made Uber turn into a bureaucratic mess. One of the VP’s of engineering called it “toil vs talent”. People who “toiled” at work, meaning doing good maintenance work, would be rewarded with good bonuses but those with “talent” would be rewarded with promotions. Of course this drove people to come up with fake new services so that they could demonstrate “talent”. This also lead to an explosion of new services that overlapped or did nothing useful. Instead of working together, groups would make new services instead of working with existing service-owners because they needed to justify writing a new service. It was sickeningly transparent. This project was one of those projects. It has no real use case because why the fuck would we want to use GPUs except to look cool on your resume. The sad thing is that the projects is overstating how well it’s being used internally. Internally people use Pinot instead of this. For all you future CTOs, consider your incentive schemes carefully and don’t be so far removed from the action that you can’t see when your org is rotting. This is what the CTO did, and like I said, it was one of his biggest failures because it gutted the engineering org. Instead of working together, every team was looking at get promotions at the expense of the company and it showed.

yakaccount4 6y ago
The culture at Uber is weird. In my opinion, there was way too much emphasis on titles. An employee's title is literally in your face in almost every internal site. This creates a lot of situations of bias because every time you get a message on slack, one of the first things you see is their pay grade. In combination with what you listed above, and the rigid, narrow pay bands, it's no wonder everyone is fishing for constant promotions.
- cellis 6y ago
  Wow. This is such a good comment. I remember when I was a junior engineer and knew more than many of the "senior" engineers at a job a few years back. It didn't matter at all, as seniority outranked sensibility. I was thinking about this a bit after reading your comment. Now, I say this as having recently negotiated pretty hard for a "Staff" title, and previously had a "Team Lead" title. I think in certain situations it makes sense to have the title authority to shoot obviously bad engineering problems down, but this has been the exception, rather than the rule, in most of my 12 years of being a software engineer. So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise. Engineering Managers or perhaps HR would be aware of the engineering votes, but engineers wouldn't, which would remove much of the implicit bias in engineering meetings. Rather, pay grades would be determined by votes, but no one would ever just "Leave it to Yakaaccount, she's the principal engineer". Everyone would have responsibility to be a solid engineer.
  - whoevercares 6y ago
    If members of the team had trusted each other enough to trust the votes that related to pay grade, you’d probably don’t need the votes already
  - underplank 6y ago
    I think its a good idea to remove implicit bias where ever you can. But doing that by making a structure that does exist be invisible is dangerous. That hierarchy does exist. Even if it's not in your face. It's human nature and it's a good thing in some situations. I think the better angle on this is to have an open culture where the discussion can happen and everyone listens and asks the questions without fear, consensus is built and those with power (because there is always someone with power) wield it sparingly. This is much harder to get to, but it's more sustainable.
  - t- 6y ago
    What you describe reminds me of the peer review system in place at a previous company I worked at. The review period was every 6 months. The way it worked was that as an individual you'd rank (and comment on) everyone you had worked with the previous 6 months based on your perception of how much value they added to the company over those previous 6 months. It could be based on some project they delivered, or them simply always answering your questions about something you needed to know to move your own projects forward, or them participating thoughtfully in company wide discussions. Once everyone had submitted their rankings, an algo would figure out everyone's final rank. The various pay level cutoffs would then be decided by eng management. It so happens that at that company everyone's title was simply 'Software Engineer'. There was no ego bs so it was a great place to work. I think how helpful you were to others being part of the review is another reason for that. There was a way to see some people's 'rank' by looking at the org chart if you really wanted to. In general swe and senior swe would be under an eng manager, if someone was directly under a director that told you something, and if someone was directly under a vp that also told you something.
  - aahortwwy 6y ago
    > So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise. I worked at a place that had a system like this. Started great, but turned toxic after a hiring spree. > Everyone would have responsibility to be a solid engineer. You would think so, but people with other intentions try to find ways to manipulate the system to achieve various strange goals. They're often successful.
- uber176 6y ago
  Post-Fowler, HR became very concerned about income inequality. They implemented this by severely compressing the range of compensation at each level. For a year or two, there was no way to meaningfully reward a high performer in her level other than to promote her. Since mid-2019 or so, the company realized this mistake and the pendulum swung back towards TK’s more Ayn Randian compensation philosophy. But the damage was done. FWIW I never saw much of a power dynamic around title. But it is definitely the most important task of an engineering manager to secure promotion-worthy projects and hand them out intelligently. Since promotion is based on impact, it’s almost entirely determined by the project charter. Difficulty or skill deployed in execution is a tertiary concern. Also given the precipitous drop in equity value, promotions with hefty raises are required just to keep people’s TC in moderate decline instead of freefall.
heliodor 6y ago
I've been wondering why the UI/UX is broken in so many ways that would be easy to fix while the engineers are busy engineering for the sake of engineering. This project, the time series database, etc. Meanwhile, the Android app can't do basic drawing of a car and a moving line on a map, for example. Your explanation of the dynamics explains the state of things quite well!
PragmaticPulp 6y ago
I wish I had understood this earlier. My past company made a push to hire from top companies like Uber for some key positions. Some of them were great people who were relieved to be out of the FAANG rat race. Others were single-mindedly focused on rewriting everything they touched with cleverly-branded project names, regardless of whether or not it made business sense. Early I on, I made a harmless comment in Slack about how one person’s pet project wasn’t a good fit for our needs so our team would be using the older, more proven solution. Later that evening the person pulled me aside, almost in tears, begging me to never say anything critical about his project in a Slack channel again. He explained that at his previous role, success or failure depends entirely on the perception of one’s personal projects and that seemingly innocent comments could tank someone’s promotion chances for years. I felt bad for him because he had clearly come out of a toxic situation. However, one of his teammates later warned me that he was keeping a journal of potentially incriminating things that I had said in Slack and a detailed log of every issue that he could find with our team’s project in case he “had to use it against me later”. I could never tell if this was a unique experience or the norm at some companies like Uber.
- hitekker 6y ago
  What happened to this person?
  - PragmaticPulp 6y ago
    His strategies worked at first, at least before he had really delivered anything of value. It started to unravel later when he couldn’t deliver on all (or really any) of his promises. I think he was operating under the assumption that he could build rapport quickly and then hire a team underneath him to get the work done. That might work at a hyper growth startup that values growth over profit, but we were a mature and profitable company looking to keep headcount reasonable. Most of his plans were so over engineered that they would have take 5-10x the engineers to actually finish on time, so they ended up being half-finished projects that requires constant on-call attention. That was the tipping point for a lot of us in the old guard. There was an exodus around that time, including myself. It’s not worth fighting those political battles day in and day out while walking on eggshells in every Slack channel.
person_of_color 6y ago
Where did you land after Uber?
hintymad 6y ago
I can concur. Most of Uber's traffic for real-time analytics was served by Uber's Elasticsearch clusters with customized querying layer. My team was also a user of the ES service. The service was sometimes unstable, but in general was fast enough for us, and scalability was never a problem. My team got paged only once from the ES team because we drove a huge spike to the team, and they ended up scanning more than 500 million records per second, for which they had to scramble to scale out. But even at that time, there was no visible degradation on our side, let alone outage. The owning team also presented their pain points, and none of the pains had to do with CPU not being fast enough. By the way, Uber already had two real-time analytics systems before AresDB. One is the aforementioned ES-based service, and the other is Pinot, which was owned by a Pinot contributor. I was in one of those so-called alignment meetings about using AresDB. Engineers from both the ES service and Pinot were there. It was a disaster. The engineers simply asked what AresDB was supposed to solve, and presented charts over charts to show that computation or lack of join operator was never a problem (because for analytics, data streams can be pre-joined), while efficient IO was. The AresDB team simply repeated that join was important, and parallel computation was critical. I left the company soon after, so I'm not sure how many critical use cases AresDB has been serving since then. Hopefully they do find some sweet spot to justify the cost of developing such system.
- alquemist 6y ago
  Not a big fan of AresDB because of the glaring IO issues. That being said, pushing 'lack of join operator was never a problem' is textbook Sapir–Whorf. 'We've trained our captive users to never ask a question that requires a join, or use convoluted workarounds if they absolutely must do so, and we are proud of it' just leaves a bad taste. I wouldn't be surprised if Pinot were to add join support in the near future...
  - hintymad 6y ago
    True that join is nice. I was just not sure if real-time join is truly necessary for Uber’s use cases, or AresDB is so much better than a query engine like Presto to justify its effort. Or more fundamentally, is GPU the solution to expensive joins?
eezurr 6y ago
This reminds of what Eddie Lampert did to Sears. He changed the company's internal structure such that internal teams has to compete against each other for bonuses and relevancy. There was a great write up that I cant find about his strategy and why it spectacularly failed.
- WrtCdEvrydy 6y ago
  Management theory now tells you to be competitive over small things. If you're doing competitiveness, do it over lunch or something else.
exhaze 6y ago
Hey, at least you didn't have to use TChannel and Hyperbahn ;)
- igmorv 6y ago
  oh my god.
totetsu 6y ago
I whish this kind of shop talk was part of more sci-fi world's world building. "No-one liked the P-class light patrol missile gunships, they were only commissioned because the space engineering devision was incentivised to deliver new projects..." on second thoughts zzzz
ethanbond 6y ago
Honestly mind blowing to me that someone could get to the CTO level of any organization - never mind something of the scale/pay grade of Uber - and think that incentive structure was going to yield anything desirable.
einpoklum 6y ago
> because why the fuck would we want to use GPUs except to look cool on your resume. Well, there are reasons, but it's true that GPUs in DBMSes is still something that's very immature. > Internally people use Pinot instead of this. Can you elaborate a bit regarding the choice of Pinot over other analytic DBMSes?

yakaccount4 6y ago

The culture at Uber is weird. In my opinion, there was way too much emphasis on titles. An employee's title is literally in your face in almost every internal site. This creates a lot of situations of bias because every time you get a message on slack, one of the first things you see is their pay grade. In combination with what you listed above, and the rigid, narrow pay bands, it's no wonder everyone is fishing for constant promotions.

cellis 6y ago
Wow. This is such a good comment. I remember when I was a junior engineer and knew more than many of the "senior" engineers at a job a few years back. It didn't matter at all, as seniority outranked sensibility. I was thinking about this a bit after reading your comment. Now, I say this as having recently negotiated pretty hard for a "Staff" title, and previously had a "Team Lead" title. I think in certain situations it makes sense to have the title authority to shoot obviously bad engineering problems down, but this has been the exception, rather than the rule, in most of my 12 years of being a software engineer. So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise. Engineering Managers or perhaps HR would be aware of the engineering votes, but engineers wouldn't, which would remove much of the implicit bias in engineering meetings. Rather, pay grades would be determined by votes, but no one would ever just "Leave it to Yakaaccount, she's the principal engineer". Everyone would have responsibility to be a solid engineer.
- whoevercares 6y ago
  If members of the team had trusted each other enough to trust the votes that related to pay grade, you’d probably don’t need the votes already
- underplank 6y ago
  I think its a good idea to remove implicit bias where ever you can. But doing that by making a structure that does exist be invisible is dangerous. That hierarchy does exist. Even if it's not in your face. It's human nature and it's a good thing in some situations. I think the better angle on this is to have an open culture where the discussion can happen and everyone listens and asks the questions without fear, consensus is built and those with power (because there is always someone with power) wield it sparingly. This is much harder to get to, but it's more sustainable.
- t- 6y ago
  What you describe reminds me of the peer review system in place at a previous company I worked at. The review period was every 6 months. The way it worked was that as an individual you'd rank (and comment on) everyone you had worked with the previous 6 months based on your perception of how much value they added to the company over those previous 6 months. It could be based on some project they delivered, or them simply always answering your questions about something you needed to know to move your own projects forward, or them participating thoughtfully in company wide discussions. Once everyone had submitted their rankings, an algo would figure out everyone's final rank. The various pay level cutoffs would then be decided by eng management. It so happens that at that company everyone's title was simply 'Software Engineer'. There was no ego bs so it was a great place to work. I think how helpful you were to others being part of the review is another reason for that. There was a way to see some people's 'rank' by looking at the org chart if you really wanted to. In general swe and senior swe would be under an eng manager, if someone was directly under a director that told you something, and if someone was directly under a vp that also told you something.
- aahortwwy 6y ago
  > So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise. I worked at a place that had a system like this. Started great, but turned toxic after a hiring spree. > Everyone would have responsibility to be a solid engineer. You would think so, but people with other intentions try to find ways to manipulate the system to achieve various strange goals. They're often successful.
uber176 6y ago
Post-Fowler, HR became very concerned about income inequality. They implemented this by severely compressing the range of compensation at each level. For a year or two, there was no way to meaningfully reward a high performer in her level other than to promote her. Since mid-2019 or so, the company realized this mistake and the pendulum swung back towards TK’s more Ayn Randian compensation philosophy. But the damage was done. FWIW I never saw much of a power dynamic around title. But it is definitely the most important task of an engineering manager to secure promotion-worthy projects and hand them out intelligently. Since promotion is based on impact, it’s almost entirely determined by the project charter. Difficulty or skill deployed in execution is a tertiary concern. Also given the precipitous drop in equity value, promotions with hefty raises are required just to keep people’s TC in moderate decline instead of freefall.

heliodor 6y ago

I've been wondering why the UI/UX is broken in so many ways that would be easy to fix while the engineers are busy engineering for the sake of engineering. This project, the time series database, etc. Meanwhile, the Android app can't do basic drawing of a car and a moving line on a map, for example. Your explanation of the dynamics explains the state of things quite well!

PragmaticPulp 6y ago

I wish I had understood this earlier. My past company made a push to hire from top companies like Uber for some key positions. Some of them were great people who were relieved to be out of the FAANG rat race. Others were single-mindedly focused on rewriting everything they touched with cleverly-branded project names, regardless of whether or not it made business sense. Early I on, I made a harmless comment in Slack about how one person’s pet project wasn’t a good fit for our needs so our team would be using the older, more proven solution. Later that evening the person pulled me aside, almost in tears, begging me to never say anything critical about his project in a Slack channel again. He explained that at his previous role, success or failure depends entirely on the perception of one’s personal projects and that seemingly innocent comments could tank someone’s promotion chances for years. I felt bad for him because he had clearly come out of a toxic situation. However, one of his teammates later warned me that he was keeping a journal of potentially incriminating things that I had said in Slack and a detailed log of every issue that he could find with our team’s project in case he “had to use it against me later”. I could never tell if this was a unique experience or the norm at some companies like Uber.

hitekker 6y ago
What happened to this person?
- PragmaticPulp 6y ago
  His strategies worked at first, at least before he had really delivered anything of value. It started to unravel later when he couldn’t deliver on all (or really any) of his promises. I think he was operating under the assumption that he could build rapport quickly and then hire a team underneath him to get the work done. That might work at a hyper growth startup that values growth over profit, but we were a mature and profitable company looking to keep headcount reasonable. Most of his plans were so over engineered that they would have take 5-10x the engineers to actually finish on time, so they ended up being half-finished projects that requires constant on-call attention. That was the tipping point for a lot of us in the old guard. There was an exodus around that time, including myself. It’s not worth fighting those political battles day in and day out while walking on eggshells in every Slack channel.

person_of_color 6y ago

Where did you land after Uber?

hintymad 6y ago

I can concur. Most of Uber's traffic for real-time analytics was served by Uber's Elasticsearch clusters with customized querying layer. My team was also a user of the ES service. The service was sometimes unstable, but in general was fast enough for us, and scalability was never a problem. My team got paged only once from the ES team because we drove a huge spike to the team, and they ended up scanning more than 500 million records per second, for which they had to scramble to scale out. But even at that time, there was no visible degradation on our side, let alone outage. The owning team also presented their pain points, and none of the pains had to do with CPU not being fast enough. By the way, Uber already had two real-time analytics systems before AresDB. One is the aforementioned ES-based service, and the other is Pinot, which was owned by a Pinot contributor. I was in one of those so-called alignment meetings about using AresDB. Engineers from both the ES service and Pinot were there. It was a disaster. The engineers simply asked what AresDB was supposed to solve, and presented charts over charts to show that computation or lack of join operator was never a problem (because for analytics, data streams can be pre-joined), while efficient IO was. The AresDB team simply repeated that join was important, and parallel computation was critical. I left the company soon after, so I'm not sure how many critical use cases AresDB has been serving since then. Hopefully they do find some sweet spot to justify the cost of developing such system.

alquemist 6y ago
Not a big fan of AresDB because of the glaring IO issues. That being said, pushing 'lack of join operator was never a problem' is textbook Sapir–Whorf. 'We've trained our captive users to never ask a question that requires a join, or use convoluted workarounds if they absolutely must do so, and we are proud of it' just leaves a bad taste. I wouldn't be surprised if Pinot were to add join support in the near future...
- hintymad 6y ago
  True that join is nice. I was just not sure if real-time join is truly necessary for Uber’s use cases, or AresDB is so much better than a query engine like Presto to justify its effort. Or more fundamentally, is GPU the solution to expensive joins?

eezurr 6y ago

This reminds of what Eddie Lampert did to Sears. He changed the company's internal structure such that internal teams has to compete against each other for bonuses and relevancy. There was a great write up that I cant find about his strategy and why it spectacularly failed.

WrtCdEvrydy 6y ago
Management theory now tells you to be competitive over small things. If you're doing competitiveness, do it over lunch or something else.

exhaze 6y ago

Hey, at least you didn't have to use TChannel and Hyperbahn ;)

igmorv 6y ago
oh my god.

totetsu 6y ago

I whish this kind of shop talk was part of more sci-fi world's world building. "No-one liked the P-class light patrol missile gunships, they were only commissioned because the space engineering devision was incentivised to deliver new projects..." on second thoughts zzzz

ethanbond 6y ago

Honestly mind blowing to me that someone could get to the CTO level of any organization - never mind something of the scale/pay grade of Uber - and think that incentive structure was going to yield anything desirable.

einpoklum 6y ago

> because why the fuck would we want to use GPUs except to look cool on your resume. Well, there are reasons, but it's true that GPUs in DBMSes is still something that's very immature. > Internally people use Pinot instead of this. Can you elaborate a bit regarding the choice of Pinot over other analytic DBMSes?