8 ms·

LedgerStore Supports Trillions of Indexes at Uber

124 points by imnot404 2y ago

dmoy 2y ago
That was a pretty cool read, thanks for posting. It always sends prickly/danger signals to my brain when I see "reads also write stuff", but this was laid out pretty clearly. Dissolved that prickly feeling pretty quickly haha.
candiddevmike 2y ago
> From a business impact perspective, operating LedgerStore is now very cost-effective due to reduced spend on DynamoDB. The estimated yearly savings are over $6 million per year. I'd like to see this broken out more. You replaced a managed service with a custom DB on top of MySQL. Surely there is a significant amount of engineering effort that was/is being spent to create/maintain this.
- stackskipton 2y ago
  Yea, 6 Million is about ~15 Engineers at Uber Total Comp average.
- BiteCode_dev 2y ago
  Managed services still require engineering efforts. I have never seen a transition to a cloud service that actually didn't end up requiring a new hire. It's the dirty little secret of our industry.
  - echelon 2y ago
    This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). When my last company transitioned to cloud, all of the on-prem folks had to additionally take on that responsibility. We then had to hire a bunch of new folks to ease the transition and manage the cloud pieces. And then there are new teams spun up for permissions and security. Your headcount requirements are not going to go down because of cloud. As for those that like to call DIY systems "weirdware": homegrown systems are expensive to engineer, but cheap to run. We 10x'd our cost going to 3rd party visibility and feature flagging versus the thinly staffed [1] homegrown systems. And those 3rd party systems also required a ton of engineering to support, the migrations weren't clean, and they missed a lot of key features that we had to upstream to the third parties! When SignalFx got bought out by Splunk, I've never seen an annual plan get re-wired so hastily. Top priority was moving to a new vendor, and that impacted every single team in the company. The only real tangible benefit to cloud that I've seen is that a team can instantly provision hardware, database, etc. resources without much planning. That's it. And is that worth it? (I don't really think so.) [1] Once built, half an engineering headcount per quarter to accommodate new features. Oncall rotation for one team that owned the systems. Our stuff was high resiliency and barely paged at all. No SEV1 or worse outages ever.
    - deanCommie 2y ago
      What a short-sighted outlook 1) "homegrown systems are expensive to engineer, but cheap to run" misses the most important part - how much does it cost to maintain, and how much of a RISK is it? Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. That's a risk. You also have to train EVERYONE you hire to learn the homegrown systems, instead of hiring people with externally transferrable skills that can come in and hit the ground running. 2) all of the on-prem folks had to additionally take on that responsibility Most transitions/migrations always end up with a period where you have both systems running. It's not surprising that at first everyone has additional responsibilities. It sounds like the "on-prem folks" either didn't have the skills to work in the cloud or refused to learn, and needed external hires. Well, that's one way to put themselves out of a job, and that would be the next step once the homegrown system is deprecated. Obviously there's a reason why "not invented here" syndrome exists. There's a reason "build vs buy" is a complex discussion because of more than just buy costs. People always prefer the home-grown thing, DIY. Engineers especially. On this site, particularly. And, also, plenty of successful businesses exist by cutting out layers and going on a shallower stack ("your margin is my opportunity"). But at the end of the day, for the vast majority of businesses, companies, and software shops, managing their own in-house infrastructure is a worse decision than paying a cloud vendor.
      - aledalgrande 2y ago
        I see both points but I have to agree with this one, especially if we’re talking small companies. A startup should outsource everything that is outside their core product (but not more than that), so they can focus on making their product top class.
      - jcelerier 2y ago
        > Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. there's so much complexity and tribal knowledge with cloud deployments that if tomorrow our cloud experts leave we're also very definitely hosed too, despite everything being documented thoroughly. I'm involved in a product that leverages a cloud-based metaverse system (Mozilla Hubs) that recently had organisational changes requiring us to change our hosting approach, and it's taking the better part of a year of work to understand its logic, for something that would have been a non-issue if homegrown & self-hosted.
  - roncesvalles 2y ago
    Eh, sure but it's not the same person. There are far more people who can "do cloud" than those who genuinely understand distributed systems well enough to build and operate a homegrown data infrastructure.
    - makmanalp 2y ago
      True, but anyone who's a cheaper hire because they can "do cloud" but not homegrown also represents a sacrifice in terms of what's you get. In terms of your operational capabilities, you'll likely be paying for that with a crisis first where everyone yells at your team because of a problem in your managed provider you can't fix, and you can't get someone from the provider on the line and helpful in a way that gets you out of it faster. So then you'll tack on add a massive yearly support retainer for a response time SLA that's still bad for your business bottom line and support that's not incentivized to prioritize you. I have stories from name-brand PaaS/IaaS companies. It's like outsourcing (I don't mean offshore, I mean any kind). It can be a reasonable thing to do if you don't pretend you're getting the same thing for less money and really actually plan around that. But people often do pretend it's the same, and the results aren't immediately apparent. And when they're finally blatantly apparent, either the guilty party has cashed out and left, or people don't have enough perspective to point to the real root cause. I'm not saying building your own infra is generally a good idea of course. It's just that beyond a certain size, for more businesses than you'd think, reliability has to be a core competency you invest in. And homegrown is a lot easier than it used to be because the offerings in terms of tooling and platforms are way better such that you can strategically craft your level of managedness for optimum cost/benefit on multiple criteria.
  - 999900000999 2y ago
    It depends on your scale. As a solo dev just being able to point my apps to firebase and not worry about it saves a ridiculous amount of time. At Uber's scale they probably have to tweak Dynamo and other services so much they might as well bring em in house. I'm a bit surprised we haven't seen a new AWS, something like Walmart Web Services.
    - wongarsu 2y ago
      On the other hand as a solo dev `sudo apt-get install postgresql` plus a short tweak of your pg_hba.conf and a quick script to run pg_dump every night is about as much work as setting up a cloud service. Managed databases provide a lot of useful features, especially if shit hits the fan. I'm not saying you shouldn't use them. But the work you put into reasonable self-hosted services and reasonable managed services often scale at a surprisingly similar pace.
      - 999900000999 2y ago
        Alright, what about Auth or other features Firebase has built in ? I still have to host it somewhere.
        BiteCode_dev 2y ago
        Auth is included in your stack, e.g: django comes with it. Plus you own the accounts, so you are not locked in. Sync is the firebase kill feature, but most apps can live without it.
        999900000999 2y ago
        For a side project it's not that serious. Firebase is ready to go without me thinking of the details. If I'm using Flutter or React, I already have a nice client side sdk to use. The 12th of never when I need to scale it I can switch to a different stack. Firebase functions allows me to do some server side data manipulation. That said, if I was at work and needed to propose something I'd probably spend more time thinking of this. But for all my side projects firebase is my goto. Let's just say we're having a competition to see who can crank out a basic crud app first. You're not beating me with Flutter and Firebase.
  - rco8786 2y ago
    Bingo. You still need your whole Ops and tooling teams. They just become Cloud Ops and Cloud Tooling.
  - devjab 2y ago
    You wish! In my experience they just load all the cloud work on to developers. Like, I’m setting up virtual networks, private endpoints, firewalls and what not, because if I don’t there won’t be horrible (I basically knew nothing about networking before I did it and I still don’t know if anything we do is correct in any manner) security but no security. It’ll bite the company in the ass eventually, or maybe it won’t, but it is what it is.
  - diob 2y ago
    I don't think it's a secret, I think the idea is it will allow you to be more resilient then a custom solution. Why? If the person(s) who made the custom solution leave, you can't as easily hire a replacement. What's more, hiring immediately effective folks becomes harder, even if those folks who made it don't leave. It's a tradeoff, but I don't think folks are keeping secrets around it.
    - otabdeveloper4 2y ago
      Not true. The hiring pool for developers that know Postgres is maybe a thousand times larger than those that know DynamoDB.
  - lanstin 2y ago
    Cloud costs more but gives you data enter as a service. Software defined everything.
  - acchow 2y ago
    What about after the transition is complete? Surely managing a… managed service… is less costly?
- stefan_ 2y ago
  When did this profession turn into a bunch of people that never want to build anything ever again? I guess I finally understand what they meant by "no one ever got fired buying IBM".
- gigatexal 2y ago
  6M in total UBER spend is it really a worthwhile chunk of change? I mean I’m here for the tech of course just curious from the cost benefit part.
  - bastawhiz 2y ago
    Six million dollars is six million dollars. Now, if that's a team of eight delivering that value, yeah, it's a great deal. If it's an org of 100 people, 6M is much less exciting. I'd suspect it's the former.
- anomaloustho 2y ago
  I think another angle that often gets overlooked is that building out these tools is a competitive edge and valuation multiplier. Harkening back to Paul Graham’s “Beating the Averages” talk about Lisp - they’ve solved problems that their competitors are also trying to solve, which could give an edge. From the valuation perspective, for that same reason, this is now an asset that can be acquired as a technology on its own - whereas using MySQL adds no intrinsic value.
jpollock 2y ago
I'm confused, why does the Trip Service talk to the issuer to place a hold? I would have expected the Payment Service to mediate all that. Particularly since "Place Hold" is another way of saying "Authorize Payment". Perhaps the lines are wrong?
- mschuster91 2y ago
  > Particularly since "Place Hold" is another way of saying "Authorize Payment". As the article explains, it's not, at least from the merchant's side (from the customer's side it is). A CC hold tells the bank "hey guys, make sure that there will be at least X dollars available when we call in the hold", and the bank can either respond with "yes, hold confirmed" or "nope, declined" (say, due to a lack of funds, wrong CVV, whatever). When the hold is confirmed, and the customer makes another transaction at another vendor that would cause the amount of open holds + executed but not paid-off transactions to go above your limit, that transaction gets blocked. And eventually, only once you as the merchant call in the hold, you will actually initiate the flow of money. Or you call in less than the held amount, and you'll only get that amount of money (usually seen in hotels and car rentals, where the hold can be significantly larger than the bill, to account for stuff like minibar expenses, room cleaning or damages to the vehicle). Or you release the hold, or it expires, and you get no money at all, and all you can do is try to create a new transaction and hope it gets approved.
  - ComputerGuru 2y ago
    But that’s what auth + capture does, though.
  - jpollock 2y ago
    Authorize Payment results in the Hold. The diagram is representing a standard Auth + Capture credit card flow. Capture contains the final transaction amount and can either be higher or lower than the Authorized amount. The odd thing is the Trip Service shouldn't know anything about the Issuer. That's the Payment Service's reason to exist (so Trip Service doesn't need to know about different payment methods). That the Trip Service knows how to talk to Visa (and Amex, and MasterCard, and Discover, and Google, and Apple) about Auth'ing a card, but doesn't know how to Capture the payment is the strange part. My expectation is that the diagram is incorrect, and the TripService talks to the PaymentService to Auth the card.
johnrob 2y ago
I’m a little confused as to what “Index” refers to here. Is it a data structure providing key lookup, or is it a single entry within one of these structures?
- tantalor 2y ago
  It has to be entries. Also confused.
- ec109685 2y ago
  It’s a materialized view (so it’s the data structure providing key lookup).
tiffanyh 2y ago
Is this at all related to the major migration from Postgres to MySQL? Previously discussed on HN (294 comments): https://news.ycombinator.com/item?id=12166585 https://news.ycombinator.com/item?id=12166585
- dmoy 2y ago
  Yea, seems like a continuation of the same fundamental problem? I don't know. As a commenter there put it - """The actual summary of the article is "The design of Postgres means that updating existing rows is inefficient compared to MySQL".""" I think this is yet another approach to dealing with the cost of updating rows?
  - wordofx 2y ago
    The reality is they had no idea what they were doing when they built Uber and made some insanely poor choices which they blamed on PostgreSQL.
ThinkBeat 2y ago
When you decide to just write your own database to store stuff in, because whatever is out there just doesn't fit your use case. you have most likely failed. You can then either write a new database that fits the use case, or take a hard look at the use case and change it to fit existing database systems. Now as a programmer I LOVE the idea of writing a database or an operating system, it would be super leet. When writing a new database system, you will make a lot of mistakes and a lot of bugs that you will no doubt encounter at inconvenient times after a lot of debugging and profiling. This is built on top of MySQL which has been battle tested (still not where I would put any critical data) Technically that you're not writing the engine but with so much functionality in two layers above MySQL you are in essence building an engine for the engine.
- gddgb 2y ago
  [dead]
- bdcravens 2y ago
  Perhaps but I think most of us have never worked on a problem of Uber's scale. The behaviors of things change spectacularly above a certain level; it's not just the same thing times N. The implicit assumption of "not invented here syndrome" is that it has been invented elsewhere already, which isn't always true.
  - convolvatron 2y ago
    whats missing from the reuse picture for things like operating systems, compilers (somwhat), and databases is systems that use a 'lego' model of components that can be put together into solutions rather than deployable systems (which can clearly be built on top). this provdies room for doing real customization without having to build all the things from ground zero, and is likely to be more robust than trying to attach functionality fully on the outside like was done here. in the database world it would be really nice to implement caching and sharding extensions inside the transactional envelope
- jonathan_landy 2y ago
  That all makes sense. On the other hand, next gen workhorses must often arise from people opting to do something new like this, rather than make do with the current gen workhorse.
normand1 2y ago
Sounds like they rebuilt QLDB? https://docs.aws.amazon.com/qldb/latest/developerguide/what-is.html https://docs.aws.amazon.com/qldb/latest/developerguide/what-...
nikolay 2y ago
Stop bragging and open-source it!
- stackskipton 2y ago
  No, please don't. Some CTO at some company I work at is going to think they are Uber scale, deploy this and I'll be stuck supporting it.
  - executethis 2y ago
    I’m sorry, my current VP of engineering is like this too. We should use falcor! Netflix stops supporting it. We should use functional programming! It became far too complex and we have abandoned it. Okay, time for a new API. Let’s use vanilla express and do everything on hard mode. Some people never learn.
  - chii 2y ago
    > I'll be stuck supporting it. which you ought to be charging a pretty penny to do so.
    - cynicalsecurity 2y ago
      Companies nowadays tend to fire people who cost them too much money. So expect supporting this monster for pennies.
    - stackskipton 2y ago
      As Ops person, they will swear someone will do it. I'll provide the hardware and it will run for a while. Then they will fire/not replace and my PagerDuty will go off constantly and I'll be stuck getting a new job.
ddoolin 2y ago
TIL that "indexes" is also a valid plural of "index", as well as "indices."
- yen223 2y ago
  The plural of "regex" is "regices"
redwood 2y ago
Feels like this would have been a good fit for zoned sharding in MongoDB -- hot shards that would age out to cold shards, never having to deal with the lack of strongly consistent secondary indexes or transactions, and not having to deal with a completely different database for the hot and cold environments.
kookamamie 2y ago
Indexes? Indices?
- antonvs 2y ago
  Both are correct. Indexes is more modern. The idea that Latin-derived words should have Latin-style plurals in English makes for pointless inconsistencies in the language.
  - kookamamie 2y ago
    One learns something new every day, as someone not native in English.
zinodaur 2y ago
Yes, writing a complete SQL DB engine like MySQL is a huge amount of work, but that is not what this article is talking about. They are wrapping another database that solves these hard problems for them. This is a fine program to write. They will be able to hire other people to maintain it. Its not rocket science
rossjudson 2y ago
Looks like you built about 10% of Spanner. That's a hard road to be on. I'm curious about what it would cost you to run the same systems on Spanner.
- kamikaz1k 2y ago
  I don't think Spanner is supposed to be cheaper than DynamoDB.
anonzzzies 2y ago
HN loves Postgres because Swiss Army knife but mysql is just much faster in most cases. We have a weird use case with data scientists and millions of tables in one db: postgres is terrible at it, mysql has no issues at all. I guess it’s all fine if you are doing an ERP or crm; I would pick Postgres. But for anything else, I would start with mysql and see how far goes without wasting your life in devops (mysql master slave and master master are built in and trivial to set up; it just works). (Sure, I hate Oracle too, but they actually did a pretty good job of both Java and mysql; i guess the 90s Borg for MS Gates and Satan for Oracle Ellison are less things now than they were)
- wordofx 2y ago
  > but mysql is just much faster in most cases What kind of cases. Since PostgreSQL wins in benchmarks I’m curious what’s faster.
  - sgarland 2y ago
    Heavy concurrency, for one. MySQL uses threads; Postgres uses forks. You can (and should) help the latter with a connection pooler, but if you then throw one in front of the MySQL instance, it wins again. Also, if you’ve designed your schema and queries around MySQL (InnoDB, specifically), you can get way faster queries. It’s a clustering index RDBMS, so range queries require far fewer reads thanks to linear read ahead, but only if your PK is k-sortable. UUIDv4 will absolutely tank the performance. It will on Postgres too, but under different circumstances, and for different reasons. Benchmarking anything is fraught with peril. There are so many variables, and the benchmark itself can be quietly selected to favor one or the other. I’d encourage you to create identically-sized instances, with your actual schema, and realistic test data.
biggestlou 2y ago
*Indices

124 points by imnot404 2y ago

dmoy 2y ago

That was a pretty cool read, thanks for posting. It always sends prickly/danger signals to my brain when I see "reads also write stuff", but this was laid out pretty clearly. Dissolved that prickly feeling pretty quickly haha.

candiddevmike 2y ago

> From a business impact perspective, operating LedgerStore is now very cost-effective due to reduced spend on DynamoDB. The estimated yearly savings are over $6 million per year. I'd like to see this broken out more. You replaced a managed service with a custom DB on top of MySQL. Surely there is a significant amount of engineering effort that was/is being spent to create/maintain this.

stackskipton 2y ago
Yea, 6 Million is about ~15 Engineers at Uber Total Comp average.
BiteCode_dev 2y ago
Managed services still require engineering efforts. I have never seen a transition to a cloud service that actually didn't end up requiring a new hire. It's the dirty little secret of our industry.
- echelon 2y ago
  This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). When my last company transitioned to cloud, all of the on-prem folks had to additionally take on that responsibility. We then had to hire a bunch of new folks to ease the transition and manage the cloud pieces. And then there are new teams spun up for permissions and security. Your headcount requirements are not going to go down because of cloud. As for those that like to call DIY systems "weirdware": homegrown systems are expensive to engineer, but cheap to run. We 10x'd our cost going to 3rd party visibility and feature flagging versus the thinly staffed [1] homegrown systems. And those 3rd party systems also required a ton of engineering to support, the migrations weren't clean, and they missed a lot of key features that we had to upstream to the third parties! When SignalFx got bought out by Splunk, I've never seen an annual plan get re-wired so hastily. Top priority was moving to a new vendor, and that impacted every single team in the company. The only real tangible benefit to cloud that I've seen is that a team can instantly provision hardware, database, etc. resources without much planning. That's it. And is that worth it? (I don't really think so.) [1] Once built, half an engineering headcount per quarter to accommodate new features. Oncall rotation for one team that owned the systems. Our stuff was high resiliency and barely paged at all. No SEV1 or worse outages ever.
  - deanCommie 2y ago
    What a short-sighted outlook 1) "homegrown systems are expensive to engineer, but cheap to run" misses the most important part - how much does it cost to maintain, and how much of a RISK is it? Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. That's a risk. You also have to train EVERYONE you hire to learn the homegrown systems, instead of hiring people with externally transferrable skills that can come in and hit the ground running. 2) all of the on-prem folks had to additionally take on that responsibility Most transitions/migrations always end up with a period where you have both systems running. It's not surprising that at first everyone has additional responsibilities. It sounds like the "on-prem folks" either didn't have the skills to work in the cloud or refused to learn, and needed external hires. Well, that's one way to put themselves out of a job, and that would be the next step once the homegrown system is deprecated. Obviously there's a reason why "not invented here" syndrome exists. There's a reason "build vs buy" is a complex discussion because of more than just buy costs. People always prefer the home-grown thing, DIY. Engineers especially. On this site, particularly. And, also, plenty of successful businesses exist by cutting out layers and going on a shallower stack ("your margin is my opportunity"). But at the end of the day, for the vast majority of businesses, companies, and software shops, managing their own in-house infrastructure is a worse decision than paying a cloud vendor.
    - aledalgrande 2y ago
      I see both points but I have to agree with this one, especially if we’re talking small companies. A startup should outsource everything that is outside their core product (but not more than that), so they can focus on making their product top class.
    - jcelerier 2y ago
      > Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. there's so much complexity and tribal knowledge with cloud deployments that if tomorrow our cloud experts leave we're also very definitely hosed too, despite everything being documented thoroughly. I'm involved in a product that leverages a cloud-based metaverse system (Mozilla Hubs) that recently had organisational changes requiring us to change our hosting approach, and it's taking the better part of a year of work to understand its logic, for something that would have been a non-issue if homegrown & self-hosted.
- roncesvalles 2y ago
  Eh, sure but it's not the same person. There are far more people who can "do cloud" than those who genuinely understand distributed systems well enough to build and operate a homegrown data infrastructure.
  - makmanalp 2y ago
    True, but anyone who's a cheaper hire because they can "do cloud" but not homegrown also represents a sacrifice in terms of what's you get. In terms of your operational capabilities, you'll likely be paying for that with a crisis first where everyone yells at your team because of a problem in your managed provider you can't fix, and you can't get someone from the provider on the line and helpful in a way that gets you out of it faster. So then you'll tack on add a massive yearly support retainer for a response time SLA that's still bad for your business bottom line and support that's not incentivized to prioritize you. I have stories from name-brand PaaS/IaaS companies. It's like outsourcing (I don't mean offshore, I mean any kind). It can be a reasonable thing to do if you don't pretend you're getting the same thing for less money and really actually plan around that. But people often do pretend it's the same, and the results aren't immediately apparent. And when they're finally blatantly apparent, either the guilty party has cashed out and left, or people don't have enough perspective to point to the real root cause. I'm not saying building your own infra is generally a good idea of course. It's just that beyond a certain size, for more businesses than you'd think, reliability has to be a core competency you invest in. And homegrown is a lot easier than it used to be because the offerings in terms of tooling and platforms are way better such that you can strategically craft your level of managedness for optimum cost/benefit on multiple criteria.
- 999900000999 2y ago
  It depends on your scale. As a solo dev just being able to point my apps to firebase and not worry about it saves a ridiculous amount of time. At Uber's scale they probably have to tweak Dynamo and other services so much they might as well bring em in house. I'm a bit surprised we haven't seen a new AWS, something like Walmart Web Services.
  - wongarsu 2y ago
    On the other hand as a solo dev `sudo apt-get install postgresql` plus a short tweak of your pg_hba.conf and a quick script to run pg_dump every night is about as much work as setting up a cloud service. Managed databases provide a lot of useful features, especially if shit hits the fan. I'm not saying you shouldn't use them. But the work you put into reasonable self-hosted services and reasonable managed services often scale at a surprisingly similar pace.
    - 999900000999 2y ago
      Alright, what about Auth or other features Firebase has built in ? I still have to host it somewhere.
      - BiteCode_dev 2y ago
        Auth is included in your stack, e.g: django comes with it. Plus you own the accounts, so you are not locked in. Sync is the firebase kill feature, but most apps can live without it.
        999900000999 2y ago
        For a side project it's not that serious. Firebase is ready to go without me thinking of the details. If I'm using Flutter or React, I already have a nice client side sdk to use. The 12th of never when I need to scale it I can switch to a different stack. Firebase functions allows me to do some server side data manipulation. That said, if I was at work and needed to propose something I'd probably spend more time thinking of this. But for all my side projects firebase is my goto. Let's just say we're having a competition to see who can crank out a basic crud app first. You're not beating me with Flutter and Firebase.
- rco8786 2y ago
  Bingo. You still need your whole Ops and tooling teams. They just become Cloud Ops and Cloud Tooling.
- devjab 2y ago
  You wish! In my experience they just load all the cloud work on to developers. Like, I’m setting up virtual networks, private endpoints, firewalls and what not, because if I don’t there won’t be horrible (I basically knew nothing about networking before I did it and I still don’t know if anything we do is correct in any manner) security but no security. It’ll bite the company in the ass eventually, or maybe it won’t, but it is what it is.
- diob 2y ago
  I don't think it's a secret, I think the idea is it will allow you to be more resilient then a custom solution. Why? If the person(s) who made the custom solution leave, you can't as easily hire a replacement. What's more, hiring immediately effective folks becomes harder, even if those folks who made it don't leave. It's a tradeoff, but I don't think folks are keeping secrets around it.
  - otabdeveloper4 2y ago
    Not true. The hiring pool for developers that know Postgres is maybe a thousand times larger than those that know DynamoDB.
- lanstin 2y ago
  Cloud costs more but gives you data enter as a service. Software defined everything.
- acchow 2y ago
  What about after the transition is complete? Surely managing a… managed service… is less costly?
stefan_ 2y ago
When did this profession turn into a bunch of people that never want to build anything ever again? I guess I finally understand what they meant by "no one ever got fired buying IBM".
gigatexal 2y ago
6M in total UBER spend is it really a worthwhile chunk of change? I mean I’m here for the tech of course just curious from the cost benefit part.
- bastawhiz 2y ago
  Six million dollars is six million dollars. Now, if that's a team of eight delivering that value, yeah, it's a great deal. If it's an org of 100 people, 6M is much less exciting. I'd suspect it's the former.
anomaloustho 2y ago
I think another angle that often gets overlooked is that building out these tools is a competitive edge and valuation multiplier. Harkening back to Paul Graham’s “Beating the Averages” talk about Lisp - they’ve solved problems that their competitors are also trying to solve, which could give an edge. From the valuation perspective, for that same reason, this is now an asset that can be acquired as a technology on its own - whereas using MySQL adds no intrinsic value.

jpollock 2y ago

I'm confused, why does the Trip Service talk to the issuer to place a hold? I would have expected the Payment Service to mediate all that. Particularly since "Place Hold" is another way of saying "Authorize Payment". Perhaps the lines are wrong?

mschuster91 2y ago
> Particularly since "Place Hold" is another way of saying "Authorize Payment". As the article explains, it's not, at least from the merchant's side (from the customer's side it is). A CC hold tells the bank "hey guys, make sure that there will be at least X dollars available when we call in the hold", and the bank can either respond with "yes, hold confirmed" or "nope, declined" (say, due to a lack of funds, wrong CVV, whatever). When the hold is confirmed, and the customer makes another transaction at another vendor that would cause the amount of open holds + executed but not paid-off transactions to go above your limit, that transaction gets blocked. And eventually, only once you as the merchant call in the hold, you will actually initiate the flow of money. Or you call in less than the held amount, and you'll only get that amount of money (usually seen in hotels and car rentals, where the hold can be significantly larger than the bill, to account for stuff like minibar expenses, room cleaning or damages to the vehicle). Or you release the hold, or it expires, and you get no money at all, and all you can do is try to create a new transaction and hope it gets approved.
- ComputerGuru 2y ago
  But that’s what auth + capture does, though.
- jpollock 2y ago
  Authorize Payment results in the Hold. The diagram is representing a standard Auth + Capture credit card flow. Capture contains the final transaction amount and can either be higher or lower than the Authorized amount. The odd thing is the Trip Service shouldn't know anything about the Issuer. That's the Payment Service's reason to exist (so Trip Service doesn't need to know about different payment methods). That the Trip Service knows how to talk to Visa (and Amex, and MasterCard, and Discover, and Google, and Apple) about Auth'ing a card, but doesn't know how to Capture the payment is the strange part. My expectation is that the diagram is incorrect, and the TripService talks to the PaymentService to Auth the card.

johnrob 2y ago

I’m a little confused as to what “Index” refers to here. Is it a data structure providing key lookup, or is it a single entry within one of these structures?

tantalor 2y ago
It has to be entries. Also confused.
ec109685 2y ago
It’s a materialized view (so it’s the data structure providing key lookup).

tiffanyh 2y ago

Is this at all related to the major migration from Postgres to MySQL? Previously discussed on HN (294 comments): https://news.ycombinator.com/item?id=12166585 https://news.ycombinator.com/item?id=12166585

dmoy 2y ago
Yea, seems like a continuation of the same fundamental problem? I don't know. As a commenter there put it - """The actual summary of the article is "The design of Postgres means that updating existing rows is inefficient compared to MySQL".""" I think this is yet another approach to dealing with the cost of updating rows?
- wordofx 2y ago
  The reality is they had no idea what they were doing when they built Uber and made some insanely poor choices which they blamed on PostgreSQL.

ThinkBeat 2y ago

When you decide to just write your own database to store stuff in, because whatever is out there just doesn't fit your use case. you have most likely failed. You can then either write a new database that fits the use case, or take a hard look at the use case and change it to fit existing database systems. Now as a programmer I LOVE the idea of writing a database or an operating system, it would be super leet. When writing a new database system, you will make a lot of mistakes and a lot of bugs that you will no doubt encounter at inconvenient times after a lot of debugging and profiling. This is built on top of MySQL which has been battle tested (still not where I would put any critical data) Technically that you're not writing the engine but with so much functionality in two layers above MySQL you are in essence building an engine for the engine.

gddgb 2y ago
[dead]
bdcravens 2y ago
Perhaps but I think most of us have never worked on a problem of Uber's scale. The behaviors of things change spectacularly above a certain level; it's not just the same thing times N. The implicit assumption of "not invented here syndrome" is that it has been invented elsewhere already, which isn't always true.
- convolvatron 2y ago
  whats missing from the reuse picture for things like operating systems, compilers (somwhat), and databases is systems that use a 'lego' model of components that can be put together into solutions rather than deployable systems (which can clearly be built on top). this provdies room for doing real customization without having to build all the things from ground zero, and is likely to be more robust than trying to attach functionality fully on the outside like was done here. in the database world it would be really nice to implement caching and sharding extensions inside the transactional envelope
jonathan_landy 2y ago
That all makes sense. On the other hand, next gen workhorses must often arise from people opting to do something new like this, rather than make do with the current gen workhorse.

normand1 2y ago

Sounds like they rebuilt QLDB? https://docs.aws.amazon.com/qldb/latest/developerguide/what-is.html https://docs.aws.amazon.com/qldb/latest/developerguide/what-...

nikolay 2y ago

Stop bragging and open-source it!

stackskipton 2y ago
No, please don't. Some CTO at some company I work at is going to think they are Uber scale, deploy this and I'll be stuck supporting it.
- executethis 2y ago
  I’m sorry, my current VP of engineering is like this too. We should use falcor! Netflix stops supporting it. We should use functional programming! It became far too complex and we have abandoned it. Okay, time for a new API. Let’s use vanilla express and do everything on hard mode. Some people never learn.
- chii 2y ago
  > I'll be stuck supporting it. which you ought to be charging a pretty penny to do so.
  - cynicalsecurity 2y ago
    Companies nowadays tend to fire people who cost them too much money. So expect supporting this monster for pennies.
  - stackskipton 2y ago
    As Ops person, they will swear someone will do it. I'll provide the hardware and it will run for a while. Then they will fire/not replace and my PagerDuty will go off constantly and I'll be stuck getting a new job.

ddoolin 2y ago

TIL that "indexes" is also a valid plural of "index", as well as "indices."

yen223 2y ago
The plural of "regex" is "regices"

redwood 2y ago

Feels like this would have been a good fit for zoned sharding in MongoDB -- hot shards that would age out to cold shards, never having to deal with the lack of strongly consistent secondary indexes or transactions, and not having to deal with a completely different database for the hot and cold environments.

kookamamie 2y ago

Indexes? Indices?

antonvs 2y ago
Both are correct. Indexes is more modern. The idea that Latin-derived words should have Latin-style plurals in English makes for pointless inconsistencies in the language.
- kookamamie 2y ago
  One learns something new every day, as someone not native in English.

zinodaur 2y ago

Yes, writing a complete SQL DB engine like MySQL is a huge amount of work, but that is not what this article is talking about. They are wrapping another database that solves these hard problems for them. This is a fine program to write. They will be able to hire other people to maintain it. Its not rocket science

rossjudson 2y ago

Looks like you built about 10% of Spanner. That's a hard road to be on. I'm curious about what it would cost you to run the same systems on Spanner.

kamikaz1k 2y ago
I don't think Spanner is supposed to be cheaper than DynamoDB.

anonzzzies 2y ago

HN loves Postgres because Swiss Army knife but mysql is just much faster in most cases. We have a weird use case with data scientists and millions of tables in one db: postgres is terrible at it, mysql has no issues at all. I guess it’s all fine if you are doing an ERP or crm; I would pick Postgres. But for anything else, I would start with mysql and see how far goes without wasting your life in devops (mysql master slave and master master are built in and trivial to set up; it just works). (Sure, I hate Oracle too, but they actually did a pretty good job of both Java and mysql; i guess the 90s Borg for MS Gates and Satan for Oracle Ellison are less things now than they were)

wordofx 2y ago
> but mysql is just much faster in most cases What kind of cases. Since PostgreSQL wins in benchmarks I’m curious what’s faster.
- sgarland 2y ago
  Heavy concurrency, for one. MySQL uses threads; Postgres uses forks. You can (and should) help the latter with a connection pooler, but if you then throw one in front of the MySQL instance, it wins again. Also, if you’ve designed your schema and queries around MySQL (InnoDB, specifically), you can get way faster queries. It’s a clustering index RDBMS, so range queries require far fewer reads thanks to linear read ahead, but only if your PK is k-sortable. UUIDv4 will absolutely tank the performance. It will on Postgres too, but under different circumstances, and for different reasons. Benchmarking anything is fraught with peril. There are so many variables, and the benchmark itself can be quietly selected to favor one or the other. I’d encourage you to create identically-sized instances, with your actual schema, and realistic test data.

biggestlou 2y ago

*Indices