6 ms·

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

bjornsing 2y ago
> Payments would not likely be a long-term bet as well How so? It’s a pretty ubiquitous problem…
kondro 2y ago
The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.
- sakjur 2y ago
  Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...
bachmeier 2y ago
> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.
- whamlastxmas 2y ago
  It’s not insulting to speculate in a conversational way around the errors we very very commonly see
- szundi 2y ago
  That was not a nice reply for a non-insult. Do you have anything to add maybe?
  - bachmeier 2y ago
    > That was not a nice reply for a non-insult. It's an insult if you dismissively explain basic things to the folks working on the project.
- inoop 2y ago
  > I'm guessing they know a lot about their costs, and you know very little. I'm curious what makes you believe the OP doesn't know about cost? They might be director-level at a large tech company with 20+ years experience for all you know... > There's little value in insulting the team members like this. I'd argue it's not insulting to question a claim (i.e. 'we saved $6MM') that is offered with little explanation.
  - qaq 2y ago
    Regardless of position at some other company it will tell you precisely 0 about this specific situation.
smokel 2y ago
At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off. There is a lot of room in between, where cost estimates are more realistic.
- szundi 2y ago
  This answer pretty much sums a lot of my experience. Of course when the guy somehow pulls this off in 2 weeks it is seen as an easy side project with proof that it is, haha
  - datadrivenangel 2y ago
    This is why incentives favor the heavy bloated enterprise approach: if it looks expensive, people feel like they got something good for their money.
- renegade-otter 2y ago
  Plenty of things can be prototyped over a weekend, but many will require months and even years to get production-ready, feature-complete, and useful, especially at scale.
cdchn 2y ago
Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.
- inoop 2y ago
  > you could get yourself 25 top-notch engineers without AI PhD Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher. Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number. And that's just the cost of labor, you haven't spun up a single machine and or sent a single byte across a wire.
  - silverquiet 2y ago
    Work from home doesn't mean that home has to be in the US.
  - cdchn 2y ago
    "and have another mil left over for metal" was the part accounting for hardware, infrastructure, etc. And you can fudge the employee salary a mil or two either way, but the point is that spending that much on a team to build something isn't infeasible or even unreasonable.
  - cthalupa 2y ago
    > Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number. Economies of scale help a bit with this for larger companies, so it's probably not quite double for Uber, but yeah, not too far off as a general rule of thumb. Probably a 75% increase on the employee facing total comp to get fairly close to the company's actual cost for the employee.
- aeyes 2y ago
  It doesn't sound like they needed to implement a new DB system for this. This is using existing features of Docstore which is Uber's own DynamoDB (sharded MySQL) which they seem to be using for almost everything.
mlrtime 2y ago
You're assuming that the team only works on this product. It is possible they are owners of a lot more than just 1 db.
inoop 2y ago
I'd be curious as well to see a more complete cost-benefit analysis, and I'd be especially interested in labor cost. We don't know how much time and head count Uber committed to this project, but I would be impressed if they were able to pull this off with fewer than 6-8 people. We can use that to get a very rough lower-bound cost estimate. For example, AWS internally uses a rule of thumb where each developer should generate about $1MM ARR (annual recurring revenue). So, if you have 20 head count, your service should bring in about $20MM annually. If Uber pulled this off with a team of ~6 engineers, by AWS logic, they should about break even. Another rule of thumb I sometimes see applied is 2x developer salary. So for example, let's assume a 7-person team of 2xSDE1, 3xSDE2, 1xSDE3, and 1xSTAFF, then according to levels.fyi that would be a total annual salary of $2.3MM. Double that, and you get $4.6MM/year to justify that team annual cost footprint, which is still less than $6MM. Of course, this is assuming a small increase in headcount to operate this new, custom data store, and does not factor in a potentially significant development and migration cost. So unless my math is completely off, it sounds to me like the cost of development, migration, and ownership is not that far off from the cost of the status quo (i.e. DynamoDb).
- Rastonbury 2y ago
  Not an engineer, but something like this takes 6-8 people working on only this for a full year?
  - inoop 2y ago
    That has been my experience, yes. You need one full-time manager, one full-time on-call/pager duty (usually a rotation), and then 4-6 people doing feature development, bug fixes, and operational stuff (e.g. applying security patches, tuning alarms, upgrading instance types, tuning auto-scaling groups, etc. etc.). Maybe you can do it a bit cheaper, e.g. with 4-6 people, but my point is that there's an on-going cost of ownership that any custom-built solution tends to incur. Amortizing that cost over many customers is essentially the entire business model of AWS :)
- shrubble 2y ago
  If the savings are 6 million per year, then in later years it should pay off since the development is a one time cost.
qaq 2y ago
If you read the article the system was a layer on top of DynamoDB they updated it to use internal product Docstore which required adding a feature to Docstore. So it's not as involved as people make it out to be. Also records are immutable which makes a lot of things way easier.
lfmunoz4 2y ago
Off the self software doesn't make sense for a company that is planning on lasting a long time. These solutions are all designed for multiple use cases. That means that there is complexity and inefficiencies that are not required for your particular problem. If you were to just focus on your problem wouldn't you just end up at an ASIC as the most optimal solution? Reason most software doesn't is 1) people like to re-invent the wheel 2) As you go start going lower level the less qualified people you can find.

bjornsing 2y ago

> Payments would not likely be a long-term bet as well How so? It’s a pretty ubiquitous problem…

kondro 2y ago

The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.

sakjur 2y ago
Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

bachmeier 2y ago

> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

whamlastxmas 2y ago
It’s not insulting to speculate in a conversational way around the errors we very very commonly see
szundi 2y ago
That was not a nice reply for a non-insult. Do you have anything to add maybe?
- bachmeier 2y ago
  > That was not a nice reply for a non-insult. It's an insult if you dismissively explain basic things to the folks working on the project.
inoop 2y ago
> I'm guessing they know a lot about their costs, and you know very little. I'm curious what makes you believe the OP doesn't know about cost? They might be director-level at a large tech company with 20+ years experience for all you know... > There's little value in insulting the team members like this. I'd argue it's not insulting to question a claim (i.e. 'we saved $6MM') that is offered with little explanation.
- qaq 2y ago
  Regardless of position at some other company it will tell you precisely 0 about this specific situation.

smokel 2y ago

At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off. There is a lot of room in between, where cost estimates are more realistic.

szundi 2y ago
This answer pretty much sums a lot of my experience. Of course when the guy somehow pulls this off in 2 weeks it is seen as an easy side project with proof that it is, haha
- datadrivenangel 2y ago
  This is why incentives favor the heavy bloated enterprise approach: if it looks expensive, people feel like they got something good for their money.
renegade-otter 2y ago
Plenty of things can be prototyped over a weekend, but many will require months and even years to get production-ready, feature-complete, and useful, especially at scale.

cdchn 2y ago

Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.

inoop 2y ago
> you could get yourself 25 top-notch engineers without AI PhD Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher. Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number. And that's just the cost of labor, you haven't spun up a single machine and or sent a single byte across a wire.
- silverquiet 2y ago
  Work from home doesn't mean that home has to be in the US.
- cdchn 2y ago
  "and have another mil left over for metal" was the part accounting for hardware, infrastructure, etc. And you can fudge the employee salary a mil or two either way, but the point is that spending that much on a team to build something isn't infeasible or even unreasonable.
- cthalupa 2y ago
  > Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number. Economies of scale help a bit with this for larger companies, so it's probably not quite double for Uber, but yeah, not too far off as a general rule of thumb. Probably a 75% increase on the employee facing total comp to get fairly close to the company's actual cost for the employee.
aeyes 2y ago
It doesn't sound like they needed to implement a new DB system for this. This is using existing features of Docstore which is Uber's own DynamoDB (sharded MySQL) which they seem to be using for almost everything.

mlrtime 2y ago

You're assuming that the team only works on this product. It is possible they are owners of a lot more than just 1 db.

inoop 2y ago

I'd be curious as well to see a more complete cost-benefit analysis, and I'd be especially interested in labor cost. We don't know how much time and head count Uber committed to this project, but I would be impressed if they were able to pull this off with fewer than 6-8 people. We can use that to get a very rough lower-bound cost estimate. For example, AWS internally uses a rule of thumb where each developer should generate about $1MM ARR (annual recurring revenue). So, if you have 20 head count, your service should bring in about $20MM annually. If Uber pulled this off with a team of ~6 engineers, by AWS logic, they should about break even. Another rule of thumb I sometimes see applied is 2x developer salary. So for example, let's assume a 7-person team of 2xSDE1, 3xSDE2, 1xSDE3, and 1xSTAFF, then according to levels.fyi that would be a total annual salary of $2.3MM. Double that, and you get $4.6MM/year to justify that team annual cost footprint, which is still less than $6MM. Of course, this is assuming a small increase in headcount to operate this new, custom data store, and does not factor in a potentially significant development and migration cost. So unless my math is completely off, it sounds to me like the cost of development, migration, and ownership is not that far off from the cost of the status quo (i.e. DynamoDb).

Rastonbury 2y ago
Not an engineer, but something like this takes 6-8 people working on only this for a full year?
- inoop 2y ago
  That has been my experience, yes. You need one full-time manager, one full-time on-call/pager duty (usually a rotation), and then 4-6 people doing feature development, bug fixes, and operational stuff (e.g. applying security patches, tuning alarms, upgrading instance types, tuning auto-scaling groups, etc. etc.). Maybe you can do it a bit cheaper, e.g. with 4-6 people, but my point is that there's an on-going cost of ownership that any custom-built solution tends to incur. Amortizing that cost over many customers is essentially the entire business model of AWS :)
shrubble 2y ago
If the savings are 6 million per year, then in later years it should pay off since the development is a one time cost.

qaq 2y ago

If you read the article the system was a layer on top of DynamoDB they updated it to use internal product Docstore which required adding a feature to Docstore. So it's not as involved as people make it out to be. Also records are immutable which makes a lot of things way easier.

lfmunoz4 2y ago

Off the self software doesn't make sense for a company that is planning on lasting a long time. These solutions are all designed for multiple use cases. That means that there is complexity and inefficiencies that are not required for your particular problem. If you were to just focus on your problem wouldn't you just end up at an ASIC as the most optimal solution? Reason most software doesn't is 1) people like to re-invent the wheel 2) As you go start going lower level the less qualified people you can find.