42 ms·

Migrating Uber's ledger data from DynamoDB to LedgerStore

328 points by gronky_ 2y ago

drexlspivey 2y ago
> Uber migrated all its payment transaction data from DynamoDB and blob storage into a new long-term solution No way they have 1 trillion transactions right?
- rco8786 2y ago
  1T "records". Any given transaction can have N records. I'm assuming this includes Uber Eats as well.
  - drexlspivey 2y ago
    Still, they have 10B rides in 2023 including Eats, say 75-100B since inception. What would be a record such that each transaction needs 10-15 on average?
    - ndr 2y ago
      Consider it might be quite de-normalized as typical at scale. Some records for the customer, some for the driver, some for the restaurant...
      - eru 2y ago
        You might even have a few more. Eg you might have a record for each stage of the meal. When it's ordered, when it's cooked, when it's delivered, etc.
    - re-thc 2y ago
      > What would be a record such that each transaction needs 10-15 on average? Does it have to be 1-dimensional? Depends exactly what payments is. There are refunds, discounts, paying e.g. drivers. There are also things like monthly subscriptions people can subscribe to for discounts / unlimited uses. Lots of things add up.
    - csomar 2y ago
      I can see that as transactions with credit cards go through lots of process (withholding, approval, charging, settling, etc..)
    - shrubble 2y ago
      They need to pay the driver and they need to handle taxes; that alone triples your estimated 100B.
    - rco8786 2y ago
      > 75-100B This seems low, off the bat. 15 years of Uber, 9 years of Uber Eats. But even just looking at my most recent trip with Uber, there are 7 different records visible on the receipt. Not including backend recordkeeping that isn't exposed to the user (driver payments, driver loan repayments, revenue recognition, internal fees/records, etc). Total trip amount, Trip fare, Booking fee, Tip, State fee, Payment #1 (trip itself), and Payment #2 (driver tip) Now consider Uber Eats where there is (at least) one record for each item in an order...plus tax, tip, etc as always. Then consider things like wait time charges, subscriptions, split charges, pending charges, chargebacks, refunds, disputes, blah blah blah. An average of 10 records per customer transaction seems entirely reasonable.
- bjornsing 2y ago
  The blog post says billions of transactions and trillions of indexes (or rather index entries I presume), if I remember correctly.
sha_r_roh 2y ago
Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?
- bjornsing 2y ago
  > Payments would not likely be a long-term bet as well How so? It’s a pretty ubiquitous problem…
- kondro 2y ago
  The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.
  - sakjur 2y ago
    Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...
- bachmeier 2y ago
  > I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.
  - whamlastxmas 2y ago
    It’s not insulting to speculate in a conversational way around the errors we very very commonly see
bjornsing 2y ago
I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/ https://www.haystackdb.dev/
- sanswork 2y ago
  You need to be doing enterprise sales not marketing. There is a lot of advice here and in general on that but you definitely need to be making calls with that type of business.
  - padjo 2y ago
    Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.
    - bjornsing 2y ago
      Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect? (I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)
      - narnarpapadaddy 2y ago
        The companies that have these types of problems all have AWS reps (or whatever vendor) that get first crack at a solution, even if their senior engineers or CTOs do some googling. Frequently discounts can be negotiated on products that aren’t a perfect fit, or companies will get early access to new products that solve their problem (AWS calls this “limited preview”). A good chunk of B2B infrastructure products like this are developed using a “golden partner” model. The first customer (or few) gets a free or reduced cost license, the developer gets a real-world scenario with real data to use to figure out what the minimal functionality actually is to be a marketable product and to work out bugs. This arrangement frequently requires a preexisting relationship and trust between both parties.
rguillebert 2y ago
So they saved $0.000006 per record, it's really about the little things...
theanirudh 2y ago
I wonder if they considered https://tigerbeetle.com https://tigerbeetle.com
- geodel 2y ago
  Would be interesting. Considering TigerBeetle is written in Zig. And Uber is probably only rare big company which has support contract with Zig foundation.
  - sa46 2y ago
    One interesting caveat is that Uber only uses the Zig toolchain, not the Zig language. https://www.uber.com/en-US/blog/bootstrapping-ubers-infrastructure-on-arm64-with-zig/ https://www.uber.com/en-US/blog/bootstrapping-ubers-infrastr...
    - bobby_the_whal 2y ago
      Only Uber can present cross-compilation as if that hasn't been done for decades. I think the newer generation is just more stupid. Must be the plastic in the water supply.
rmccue 2y ago
Original story looks to be https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...
deadbabe 2y ago
So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?
- Galanwe 2y ago
  Employees are constantly saving cost or adding value, that's what they are paid for.
- zinglersen 2y ago
  If a project fails, do you pay for the loss since you want a share of the profits as well?
  - deadbabe 2y ago
    Someone probably gets fired so I guess someone does pay the ultimate price.
    - zinglersen 2y ago
      Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price. "The metaverse division has now lost more than $45 billion since the end of 2020" Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?
      - HeatrayEnjoyer 2y ago
        The "risk takers" are not taking at any risk at all. What's the chance they end up on the street, or even suffer personal financial stress about their life? That they will have to move, sell their car, home, etc. It's 0%.
        zinglersen 2y ago
        What.. they are taking a lot of risk... But I guess we first have to agree on "who" we are taking about - is it the company itself or the owner / shareholders ? Back to your question, yes that could happen in several different cases. But of course the risk/benefit is not split 50/50 (nor 0 risk, 100 upside, as you said), in reality the future outcome depends on both internal and external events. Even the richest(?) man in the world was relatively close to loosing it all; Musk, who had $200 million in cash at one point, invested “his last cent in his businesses” and said in a 2010 divorce proceeding, “About four months ago, I ran out of cash.” Musk told the New York Times https://www.cnbc.com/2017/04/27/the-crucial-decision-teslas-elon-musk-had-to-make-when-he-was-broke.html https://www.cnbc.com/2017/04/27/the-crucial-decision-teslas-... https://archive.nytimes.com/dealbook.nytimes.com/2010/06/22/sorkin-elon-musk-of-paypal-and-tesla-fame-is-broke/ https://archive.nytimes.com/dealbook.nytimes.com/2010/06/22/...
deleted 2y ago
[deleted]
drpotato 2y ago
The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions-of-indexes https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...
- intunderflow 2y ago
  Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?
  - gronky_ 2y ago
    Yes, that’s definitely the main reason. It’s called “burying the lede”. Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration
    - dboreham 2y ago
      TaaS : title as a service
      - k1t 2y ago
        People have done this, eg https://www.reddit.com/r/GrowthHacking/comments/k20g42/ai_tool_for_choosing_the_best_hacker_news_title/ https://www.reddit.com/r/GrowthHacking/comments/k20g42/ai_to... However that appears to be defunct now
    - alexchantavy 2y ago
      I'm usually guilty of this. The hands-on person involved in a highly technical project gets excited and bogged down in the details of the project that they end up not being the most compelling storyteller about it.
_xnmw 2y ago
I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a-single-server https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
- sanderjd 2y ago
  Sometimes things just aren't nails, even when you have a really good hammer.
- kondro 2y ago
  Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.
  - bayindirh 2y ago
    If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
    - zaphirplane 2y ago
      > or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
      - bayindirh 2y ago
        These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal. Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand. These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.
benterix 2y ago
I read the article so I roughly know what LedgerStore is - but I have no idea where it is hosted.
- tiew9Vii 2y ago
  From one of the original sources linked in this thread > LSG promised shorter indexing lag (i.e., time between when a record is written and its secondary index is created). Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...
xiwenc 2y ago
Is this another outlier when you reach certain scale, it’s more beneficial to roll your own? Pretty amazing what Uber has to deal with. Also it’s not very clear from the original articles, what is the new total “cost of ownership” of this new refactored service. Like now they need to manage their own databases and the storage backing them. Or did i miss it?
- crabbone 2y ago
  I worked for a company which used Redis at the prototyping phase, but then wrote own database to improve performance and resilience. The company wasn't selling an end-user facing product, the product was a distributed filesystem. My take on this is that most companies don't have the expertise to build systems like databases, and even if the costs would otherwise suggest such a development as desirable would be simply afraid of doing it.
ForHackernews 2y ago
Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.
- sanderjd 2y ago
  I'm not sure if they have regulatory obligations to keep them, or what, but it still seems like you could back them up to cold storage after a reasonable period of time.
- robertlagrant 2y ago
  It might just be an internal policy to cover all the crazy combinations of regs the world over. They might just say 10/20/100 years is their policy, now figure out how to store it.
- moooo99 2y ago
  Payment information is often subject to pretty strict regulatory requirements, including archival durations. Having to keep all the original information for 10 years is not entirely uncommon.
- crabbone 2y ago
  In systems that deal with money, money-related data is virtually never deleted. The reason is the fear that deletion can be exploited somehow in the future, rather than the old data being actionable. For example, if a customer registers with the name of a deleted customer, which will resurface some "unfinished" transactions or rules associated with the older version of the "same" customer that haven't been properly deleted but appeared to be deleted for a while. Also, in general, deletion is very difficult because money doesn't just disappear. You'd need some sort of compaction (think: Git squash) rather than deletion to be able to balance the system's books... but then you'd be filling the system with fake transactions... From my experience from working with these kinds of systems, the typical solution is to label entities with active/inactive labels to substitute deletion. But entities never go away.
  - kobalsky 2y ago
washywashy 2y ago
I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some critical fix.
- gtirloni 2y ago
  If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.
  - vertis 2y ago
    Companies this size almost certainly have different terms of use. I worked for a smaller, but still ASX200 company that had a custom contract, and assigned staff that would drop by 2-3 days a month. Of specific note was that if AWS wanted to stop doing business with us they had to give at least X notice (from memory that was 12 months for us). For our risk profile this was more than enough time to migrate off any AWS' proprietary technology. That makes it worth less to avoid exposure.
  - scarface_74 2y ago
    This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.
    - gtirloni 2y ago
      *> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?
debarshri 2y ago
There was an era around 2015, when all the cool tech companies like netflix, spotify, soundcloud, uber and others were building alot of infrastructure and database tools. Nowadays, engineers often talk in AWS/Cloud terminologies. It is breathe of fresh air to see that orgs are still building tools like that.
augunrik 2y ago
Is there some information on why they need to store this much data for immediate retrieval? And why is it so much?
Antony90807 2y ago
Wow crazy amount of work went into this. Well done
influx 2y ago
I would gladly pay 6 million/year to not be on call, and have to worry about things like bios and ssd firmware ever again.
- overstay8930 2y ago
  Yup, every time we bring up migrations of this sort it pretty much stops when we start talking about maintenance and on-call schedules. Turns out it costs more to DIY when people start quitting because they have to work during a holiday.
- geodel 2y ago
  Thats a great situation to be in when one can spend 6 million even when there was some chance to save. I tried same for ready to eat meal everyday to save me from potential kitchen disasters but sadly numbers didn't work out.
  - influx 2y ago
    You're not saving money, controlling your own destiny sure. That's worth something, maybe even more than 6 million, but I was a SRE at Uber who had to be oncall for systems like this, believe it or not, people like me aren't free either :)
- tedd4u 2y ago
  My "favorite" non-cloud issue was dying/dead RAID card batteries in DB hosts (to preserve unflushed cache in RAM on the card in case of power failure).
- ec109685 2y ago
  They didn’t say they are running this on prem.
  - tuananh 2y ago
    > Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.
boringg 2y ago
How much did the migration effort cost?
geodel 2y ago
More power to them. At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS. After carefully evaluating our strategic course of action answer always is AWS. Its only team who propose alternative they have to justify rigorously how come they differ in conclusion.
- dboreham 2y ago
  My Amazon stock thanks you.
- tonyhart7 2y ago
  "At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS" Yeah until those bill come, They would consider alternative
- cess11 2y ago
  Not if you're in the EU, due to, among other things, Schrems II.
  - graemep 2y ago
    AFAIK Schrems II prevents transfers of data to the US. AWS has datacentres around the world, including multiple locations in the EU.
    - cess11 2y ago
      Where did you learn that? Schrems II prohibits transfer of personal information to companies reachable by the CLOUD act.
deleted 2y ago
[deleted]
ramesh31 2y ago
Another victim of the "Great Normalization", i.e. that entire generation of garbage tech debt generated during the 2010s that was built on NoSQL stores that never should have been, is now coming due. You could probably make an entire consulting business out of migrating these things to MySQL.
- cgh 2y ago
  This is exactly the comment I came here to make. The NoSql technical debt accretes like dead leaves blocking a sewer drain. Eventually someone has to wade in and normalize…the sewer grate, I guess. Okay, it’s not a perfect analogy.
- riku_iki 2y ago
  > . You could probably make an entire consulting business out of migrating these things to MySQL. I think it will be very complex task to run MySQL for 1PB 1T transactions..
  - deleted 2y ago
    [deleted]
- collyw 2y ago
  This seemed so obvious at the time, you were throwing away so many useful features for the promise of "web scale". I argued with many developers at the time, but they insisted that NoSQL (Mongo) needed to be used.
jcims 2y ago
Every time I've ever used DynamoDB it cost way more than I would have ever expected.
- alexey-salmin 2y ago
  Yes
schmidt_fifty 2y ago
[dead]
ledgerdev 2y ago
Say you wanted to build an app on a database like LedgerStore but at much smaller scale, what are the best open source options out there right now?
- superzamp 2y ago
  We have a pretty minimal setup at formancehq/ledger[1] that uses pg as a storage backend, and comes with a programmability layer to model complex transactions (e.g. sourcing $100 from three accounts in cascade). [1] https://github.com/formancehq/ledger https://github.com/formancehq/ledger
qwertyuiop_ 2y ago
Assuming there are a minimum of two teams a total of 20 maintaining this in-house software, I gave 250k as cost per engineer (salary plus health and other benefit costs to the company). Thats $5 million right there. I am estimating lowest range. Thats why Amazon calls these efforts undifferentiated heavy lifting. is there a slight premium to pay than rolling your own and maintaining yes. Its worth all the trouble and security and management overhead into rolling your own.
xyst 2y ago
Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.
- cj 2y ago
  $6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engineering-manager?country=254 https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/uber/salaries/software-engineer?country=254 https://www.levels.fyi/companies/uber/salaries/software-engi...
  - booi 2y ago
    That’s what I was thinking and fully loaded cost at least 35% more than their salary. Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it. Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.
  - dangus 2y ago
    Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.
  - leoqa 2y ago
    Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.
foota 2y ago
The article states that they already had an in house solution for cold data, so one of the benefits they claim is simplifying by moving to one system for both hot and cold data.
citizenpaul 2y ago
I think there is some reckoning of cloud service providers coming(assuming logical actors...). I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. They hired some out of school data scientist to do reports and they were doing crazy ineffective things with the tiny dataset. Wanted me to fix it for pennies tomorrow and I declined.
- remus 2y ago
  Not that I disagree with your overall point, but I don't think this > I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. Is a good example. It's just a badly architected system, and you'd have exactly the same problem if you were running the same thing on a massively over provisioned on premise db.
  - citizenpaul 2y ago
    I think its a perfect example that cloud providers are profiting on a unsustainable scheme. They rely on their services papering over (literally with money) lack of skill/knowlege of their customers. This tactic either kills the customer or creates someone that is desperate to no longer be your customer.
    - bobby_the_whal 2y ago
      People are getting dumber by the hour. It's sustainable. When I design computing systems, I am like "what's the point?", if there is nobody with a brain around to maintain it. The cloud allows companies to have services they could never ever maintain themselves long after I am dead. We are going full Idiocracy from what I can tell. You sound way too optimistic, but I am guessing you are still young.
PeterZaitsev 2y ago
I think this is fantastic illustration of how expensive proprietary cloud based data stores can be... and what it is feasible to migrate from them to something else.
benced 2y ago
$6M... isn't that much?
SkyMarshal 2y ago
It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021: https://www.uber.com/en-US/blog/dynamodb-to-docstore-migration/ https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati... [1]:https://github.com/uber https://github.com/uber
- PeterZaitsev 2y ago
  Yeah. This looks like some internal solution. In general Uber seems to be high on "not invented here" scale - they like to conclude no existing Open Source solutions are good enough for them and they need to build their own... this is different from Facebook approach for example which chose to made MySQL better by adding MyRocks/RocksDB to it and keep them Open Source.
  - vertis 2y ago
    It's a weird world where Facebook/Meta is becoming a small bastion of hope. Llama 2/3 being an example of bucking the trend of going closed source for LLM models. Granted it's not quite in the same calibre as OpenAI/Claude, and the real test is when it is and they still release it.
  - rubiquity 2y ago
    Don't worry it's 99% likely LedgerStore is built on top of MySQL.
alexey-salmin 2y ago
I don't know about economics of this particular project but damn dynamodb is expensive. At some point I was thinking that everyone else was just using it wrong, doing scans and queries instead of point-wise lookups into pre-computed tables. It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.
- kikimora 2y ago
  Why? 120 usd per 100 WCU per year, 30 usd per 100 RCU, does not sound expensive. 1 RCU reads up to 4Kb, to read 100 MBs you would need 100 000 RCU which would cost 30 000 usd/year, or 2500/year. Unless my math is off I don’t think anything come close in terms of price.
pojzon 2y ago
You look at stuff like that and think about „how much talent is wasted on pointless things that help noone in the world while getting paid heaps for nothing” We could accomplish everything if ppl stopped wasting time on pointless tasks.
otterley 2y ago
Does anyone know whether Uber considered Amazon QLDB for the implementation? Seems like it might have been a good fit, at first blush.
yazaddaruvala 2y ago
Reading the article it’s clear pretty quickly that Uber was using DynamoDB poorly. It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions. It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.
- PartiallyTyped 2y ago
  I don't understand why they needed 2 weeks of immutable transactions in Dynamo. Could anyone give any hints?
- cced 2y ago
  Can you post some references to this pattern?
  - yazaddaruvala 2y ago
    Mostly just look into “archiving into data warehouse” type patterns. You can also look into “realtime data warehousing”. Meanwhile, it’s so common, AWS built it into their product. https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dynamodb-zero-etl-integration-redshift/ https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dy...
awinter-py 2y ago
they do 2 billion rides per quarter what are the 'trillions' here? ^ also that translates to ~1000 transactions per second with some assumptions; have never understood why they care so much about infra scaling 1000 tps is like 1 box

328 points by gronky_ 2y ago

> Uber migrated all its payment transaction data from DynamoDB and blob storage into a new long-term solution No way they have 1 trillion transactions right?

rco8786 2y ago
1T "records". Any given transaction can have N records. I'm assuming this includes Uber Eats as well.
- drexlspivey 2y ago
  Still, they have 10B rides in 2023 including Eats, say 75-100B since inception. What would be a record such that each transaction needs 10-15 on average?
  - ndr 2y ago
    Consider it might be quite de-normalized as typical at scale. Some records for the customer, some for the driver, some for the restaurant...
    - eru 2y ago
      You might even have a few more. Eg you might have a record for each stage of the meal. When it's ordered, when it's cooked, when it's delivered, etc.
  - re-thc 2y ago
    > What would be a record such that each transaction needs 10-15 on average? Does it have to be 1-dimensional? Depends exactly what payments is. There are refunds, discounts, paying e.g. drivers. There are also things like monthly subscriptions people can subscribe to for discounts / unlimited uses. Lots of things add up.
  - csomar 2y ago
    I can see that as transactions with credit cards go through lots of process (withholding, approval, charging, settling, etc..)
  - shrubble 2y ago
    They need to pay the driver and they need to handle taxes; that alone triples your estimated 100B.
  - rco8786 2y ago
    > 75-100B This seems low, off the bat. 15 years of Uber, 9 years of Uber Eats. But even just looking at my most recent trip with Uber, there are 7 different records visible on the receipt. Not including backend recordkeeping that isn't exposed to the user (driver payments, driver loan repayments, revenue recognition, internal fees/records, etc). Total trip amount, Trip fare, Booking fee, Tip, State fee, Payment #1 (trip itself), and Payment #2 (driver tip) Now consider Uber Eats where there is (at least) one record for each item in an order...plus tax, tip, etc as always. Then consider things like wait time charges, subscriptions, split charges, pending charges, chargebacks, refunds, disputes, blah blah blah. An average of 10 records per customer transaction seems entirely reasonable.
bjornsing 2y ago
The blog post says billions of transactions and trillions of indexes (or rather index entries I presume), if I remember correctly.

sha_r_roh 2y ago

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

bjornsing 2y ago
> Payments would not likely be a long-term bet as well How so? It’s a pretty ubiquitous problem…
kondro 2y ago
The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.
- sakjur 2y ago
  Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...
bachmeier 2y ago
> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.
- whamlastxmas 2y ago
  It’s not insulting to speculate in a conversational way around the errors we very very commonly see

bjornsing 2y ago

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/ https://www.haystackdb.dev/

sanswork 2y ago
You need to be doing enterprise sales not marketing. There is a lot of advice here and in general on that but you definitely need to be making calls with that type of business.
- padjo 2y ago
  Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.
  - bjornsing 2y ago
    Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect? (I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)
    - narnarpapadaddy 2y ago
      The companies that have these types of problems all have AWS reps (or whatever vendor) that get first crack at a solution, even if their senior engineers or CTOs do some googling. Frequently discounts can be negotiated on products that aren’t a perfect fit, or companies will get early access to new products that solve their problem (AWS calls this “limited preview”). A good chunk of B2B infrastructure products like this are developed using a “golden partner” model. The first customer (or few) gets a free or reduced cost license, the developer gets a real-world scenario with real data to use to figure out what the minimal functionality actually is to be a marketable product and to work out bugs. This arrangement frequently requires a preexisting relationship and trust between both parties.

rguillebert 2y ago

So they saved $0.000006 per record, it's really about the little things...

theanirudh 2y ago

I wonder if they considered https://tigerbeetle.com https://tigerbeetle.com

geodel 2y ago
Would be interesting. Considering TigerBeetle is written in Zig. And Uber is probably only rare big company which has support contract with Zig foundation.
- sa46 2y ago
  One interesting caveat is that Uber only uses the Zig toolchain, not the Zig language. https://www.uber.com/en-US/blog/bootstrapping-ubers-infrastructure-on-arm64-with-zig/ https://www.uber.com/en-US/blog/bootstrapping-ubers-infrastr...
  - bobby_the_whal 2y ago
    Only Uber can present cross-compilation as if that hasn't been done for decades. I think the newer generation is just more stupid. Must be the plastic in the water supply.

rmccue 2y ago

Original story looks to be https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

deadbabe 2y ago

So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?

Galanwe 2y ago
Employees are constantly saving cost or adding value, that's what they are paid for.
zinglersen 2y ago
If a project fails, do you pay for the loss since you want a share of the profits as well?
- deadbabe 2y ago
  Someone probably gets fired so I guess someone does pay the ultimate price.
  - zinglersen 2y ago
    Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price. "The metaverse division has now lost more than $45 billion since the end of 2020" Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?
    - HeatrayEnjoyer 2y ago
      The "risk takers" are not taking at any risk at all. What's the chance they end up on the street, or even suffer personal financial stress about their life? That they will have to move, sell their car, home, etc. It's 0%.
      - zinglersen 2y ago
        What.. they are taking a lot of risk... But I guess we first have to agree on "who" we are taking about - is it the company itself or the owner / shareholders ? Back to your question, yes that could happen in several different cases. But of course the risk/benefit is not split 50/50 (nor 0 risk, 100 upside, as you said), in reality the future outcome depends on both internal and external events. Even the richest(?) man in the world was relatively close to loosing it all; Musk, who had $200 million in cash at one point, invested “his last cent in his businesses” and said in a 2010 divorce proceeding, “About four months ago, I ran out of cash.” Musk told the New York Times https://www.cnbc.com/2017/04/27/the-crucial-decision-teslas-elon-musk-had-to-make-when-he-was-broke.html https://www.cnbc.com/2017/04/27/the-crucial-decision-teslas-... https://archive.nytimes.com/dealbook.nytimes.com/2010/06/22/sorkin-elon-musk-of-paypal-and-tesla-fame-is-broke/ https://archive.nytimes.com/dealbook.nytimes.com/2010/06/22/...

deleted 2y ago

[deleted]

drpotato 2y ago

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions-of-indexes https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

intunderflow 2y ago
Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?
- gronky_ 2y ago
  Yes, that’s definitely the main reason. It’s called “burying the lede”. Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration
  - dboreham 2y ago
    TaaS : title as a service
    - k1t 2y ago
      People have done this, eg https://www.reddit.com/r/GrowthHacking/comments/k20g42/ai_tool_for_choosing_the_best_hacker_news_title/ https://www.reddit.com/r/GrowthHacking/comments/k20g42/ai_to... However that appears to be defunct now
  - alexchantavy 2y ago
    I'm usually guilty of this. The hands-on person involved in a highly technical project gets excited and bogged down in the details of the project that they end up not being the most compelling storyteller about it.

_xnmw 2y ago

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a-single-server https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

sanderjd 2y ago
Sometimes things just aren't nails, even when you have a really good hammer.
kondro 2y ago
Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.
- bayindirh 2y ago
  If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
  - zaphirplane 2y ago
    > or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
    - bayindirh 2y ago
      These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal. Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand. These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.

benterix 2y ago

I read the article so I roughly know what LedgerStore is - but I have no idea where it is hosted.

tiew9Vii 2y ago
From one of the original sources linked in this thread > LSG promised shorter indexing lag (i.e., time between when a record is written and its secondary index is created). Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers. https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-ledgerstore/ https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

xiwenc 2y ago

Is this another outlier when you reach certain scale, it’s more beneficial to roll your own? Pretty amazing what Uber has to deal with. Also it’s not very clear from the original articles, what is the new total “cost of ownership” of this new refactored service. Like now they need to manage their own databases and the storage backing them. Or did i miss it?

crabbone 2y ago
I worked for a company which used Redis at the prototyping phase, but then wrote own database to improve performance and resilience. The company wasn't selling an end-user facing product, the product was a distributed filesystem. My take on this is that most companies don't have the expertise to build systems like databases, and even if the costs would otherwise suggest such a development as desirable would be simply afraid of doing it.

ForHackernews 2y ago

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

sanderjd 2y ago
I'm not sure if they have regulatory obligations to keep them, or what, but it still seems like you could back them up to cold storage after a reasonable period of time.
robertlagrant 2y ago
It might just be an internal policy to cover all the crazy combinations of regs the world over. They might just say 10/20/100 years is their policy, now figure out how to store it.
moooo99 2y ago
Payment information is often subject to pretty strict regulatory requirements, including archival durations. Having to keep all the original information for 10 years is not entirely uncommon.
crabbone 2y ago
In systems that deal with money, money-related data is virtually never deleted. The reason is the fear that deletion can be exploited somehow in the future, rather than the old data being actionable. For example, if a customer registers with the name of a deleted customer, which will resurface some "unfinished" transactions or rules associated with the older version of the "same" customer that haven't been properly deleted but appeared to be deleted for a while. Also, in general, deletion is very difficult because money doesn't just disappear. You'd need some sort of compaction (think: Git squash) rather than deletion to be able to balance the system's books... but then you'd be filling the system with fake transactions... From my experience from working with these kinds of systems, the typical solution is to label entities with active/inactive labels to substitute deletion. But entities never go away.
- kobalsky 2y ago

washywashy 2y ago

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some critical fix.

gtirloni 2y ago
If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.
- vertis 2y ago
  Companies this size almost certainly have different terms of use. I worked for a smaller, but still ASX200 company that had a custom contract, and assigned staff that would drop by 2-3 days a month. Of specific note was that if AWS wanted to stop doing business with us they had to give at least X notice (from memory that was 12 months for us). For our risk profile this was more than enough time to migrate off any AWS' proprietary technology. That makes it worth less to avoid exposure.
- scarface_74 2y ago
  This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.
  - gtirloni 2y ago
    *> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?

debarshri 2y ago

There was an era around 2015, when all the cool tech companies like netflix, spotify, soundcloud, uber and others were building alot of infrastructure and database tools. Nowadays, engineers often talk in AWS/Cloud terminologies. It is breathe of fresh air to see that orgs are still building tools like that.

augunrik 2y ago

Is there some information on why they need to store this much data for immediate retrieval? And why is it so much?

Antony90807 2y ago

Wow crazy amount of work went into this. Well done

influx 2y ago

I would gladly pay 6 million/year to not be on call, and have to worry about things like bios and ssd firmware ever again.

overstay8930 2y ago
Yup, every time we bring up migrations of this sort it pretty much stops when we start talking about maintenance and on-call schedules. Turns out it costs more to DIY when people start quitting because they have to work during a holiday.
geodel 2y ago
Thats a great situation to be in when one can spend 6 million even when there was some chance to save. I tried same for ready to eat meal everyday to save me from potential kitchen disasters but sadly numbers didn't work out.
- influx 2y ago
  You're not saving money, controlling your own destiny sure. That's worth something, maybe even more than 6 million, but I was a SRE at Uber who had to be oncall for systems like this, believe it or not, people like me aren't free either :)
tedd4u 2y ago
My "favorite" non-cloud issue was dying/dead RAID card batteries in DB hosts (to preserve unflushed cache in RAM on the card in case of power failure).
ec109685 2y ago
They didn’t say they are running this on prem.
- tuananh 2y ago
  > Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.

boringg 2y ago

How much did the migration effort cost?

geodel 2y ago

More power to them. At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS. After carefully evaluating our strategic course of action answer always is AWS. Its only team who propose alternative they have to justify rigorously how come they differ in conclusion.

dboreham 2y ago
My Amazon stock thanks you.
tonyhart7 2y ago
"At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS" Yeah until those bill come, They would consider alternative
cess11 2y ago
Not if you're in the EU, due to, among other things, Schrems II.
- graemep 2y ago
  AFAIK Schrems II prevents transfers of data to the US. AWS has datacentres around the world, including multiple locations in the EU.
  - cess11 2y ago
    Where did you learn that? Schrems II prohibits transfer of personal information to companies reachable by the CLOUD act.

deleted 2y ago

[deleted]

ramesh31 2y ago

Another victim of the "Great Normalization", i.e. that entire generation of garbage tech debt generated during the 2010s that was built on NoSQL stores that never should have been, is now coming due. You could probably make an entire consulting business out of migrating these things to MySQL.

cgh 2y ago
This is exactly the comment I came here to make. The NoSql technical debt accretes like dead leaves blocking a sewer drain. Eventually someone has to wade in and normalize…the sewer grate, I guess. Okay, it’s not a perfect analogy.
riku_iki 2y ago
> . You could probably make an entire consulting business out of migrating these things to MySQL. I think it will be very complex task to run MySQL for 1PB 1T transactions..
- deleted 2y ago
  [deleted]
collyw 2y ago
This seemed so obvious at the time, you were throwing away so many useful features for the promise of "web scale". I argued with many developers at the time, but they insisted that NoSQL (Mongo) needed to be used.

jcims 2y ago

Every time I've ever used DynamoDB it cost way more than I would have ever expected.

alexey-salmin 2y ago
Yes

schmidt_fifty 2y ago

[dead]

ledgerdev 2y ago

Say you wanted to build an app on a database like LedgerStore but at much smaller scale, what are the best open source options out there right now?

superzamp 2y ago
We have a pretty minimal setup at formancehq/ledger[1] that uses pg as a storage backend, and comes with a programmability layer to model complex transactions (e.g. sourcing $100 from three accounts in cascade). [1] https://github.com/formancehq/ledger https://github.com/formancehq/ledger

qwertyuiop_ 2y ago

Assuming there are a minimum of two teams a total of 20 maintaining this in-house software, I gave 250k as cost per engineer (salary plus health and other benefit costs to the company). Thats $5 million right there. I am estimating lowest range. Thats why Amazon calls these efforts undifferentiated heavy lifting. is there a slight premium to pay than rolling your own and maintaining yes. Its worth all the trouble and security and management overhead into rolling your own.

xyst 2y ago

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

cj 2y ago
$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engineering-manager?country=254 https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/uber/salaries/software-engineer?country=254 https://www.levels.fyi/companies/uber/salaries/software-engi...
- booi 2y ago
  That’s what I was thinking and fully loaded cost at least 35% more than their salary. Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it. Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.
- dangus 2y ago
  Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.
- leoqa 2y ago
  Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.

foota 2y ago

The article states that they already had an in house solution for cold data, so one of the benefits they claim is simplifying by moving to one system for both hot and cold data.

citizenpaul 2y ago

I think there is some reckoning of cloud service providers coming(assuming logical actors...). I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. They hired some out of school data scientist to do reports and they were doing crazy ineffective things with the tiny dataset. Wanted me to fix it for pennies tomorrow and I declined.

remus 2y ago
Not that I disagree with your overall point, but I don't think this > I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. Is a good example. It's just a badly architected system, and you'd have exactly the same problem if you were running the same thing on a massively over provisioned on premise db.
- citizenpaul 2y ago
  I think its a perfect example that cloud providers are profiting on a unsustainable scheme. They rely on their services papering over (literally with money) lack of skill/knowlege of their customers. This tactic either kills the customer or creates someone that is desperate to no longer be your customer.
  - bobby_the_whal 2y ago
    People are getting dumber by the hour. It's sustainable. When I design computing systems, I am like "what's the point?", if there is nobody with a brain around to maintain it. The cloud allows companies to have services they could never ever maintain themselves long after I am dead. We are going full Idiocracy from what I can tell. You sound way too optimistic, but I am guessing you are still young.

PeterZaitsev 2y ago

I think this is fantastic illustration of how expensive proprietary cloud based data stores can be... and what it is feasible to migrate from them to something else.

benced 2y ago

$6M... isn't that much?

SkyMarshal 2y ago

It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021: https://www.uber.com/en-US/blog/dynamodb-to-docstore-migration/ https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati... [1]:https://github.com/uber https://github.com/uber

PeterZaitsev 2y ago
Yeah. This looks like some internal solution. In general Uber seems to be high on "not invented here" scale - they like to conclude no existing Open Source solutions are good enough for them and they need to build their own... this is different from Facebook approach for example which chose to made MySQL better by adding MyRocks/RocksDB to it and keep them Open Source.
- vertis 2y ago
  It's a weird world where Facebook/Meta is becoming a small bastion of hope. Llama 2/3 being an example of bucking the trend of going closed source for LLM models. Granted it's not quite in the same calibre as OpenAI/Claude, and the real test is when it is and they still release it.
- rubiquity 2y ago
  Don't worry it's 99% likely LedgerStore is built on top of MySQL.

alexey-salmin 2y ago

I don't know about economics of this particular project but damn dynamodb is expensive. At some point I was thinking that everyone else was just using it wrong, doing scans and queries instead of point-wise lookups into pre-computed tables. It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.

kikimora 2y ago
Why? 120 usd per 100 WCU per year, 30 usd per 100 RCU, does not sound expensive. 1 RCU reads up to 4Kb, to read 100 MBs you would need 100 000 RCU which would cost 30 000 usd/year, or 2500/year. Unless my math is off I don’t think anything come close in terms of price.

pojzon 2y ago

You look at stuff like that and think about „how much talent is wasted on pointless things that help noone in the world while getting paid heaps for nothing” We could accomplish everything if ppl stopped wasting time on pointless tasks.

otterley 2y ago

Does anyone know whether Uber considered Amazon QLDB for the implementation? Seems like it might have been a good fit, at first blush.

yazaddaruvala 2y ago

Reading the article it’s clear pretty quickly that Uber was using DynamoDB poorly. It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions. It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.

PartiallyTyped 2y ago
I don't understand why they needed 2 weeks of immutable transactions in Dynamo. Could anyone give any hints?
cced 2y ago
Can you post some references to this pattern?
- yazaddaruvala 2y ago
  Mostly just look into “archiving into data warehouse” type patterns. You can also look into “realtime data warehousing”. Meanwhile, it’s so common, AWS built it into their product. https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dynamodb-zero-etl-integration-redshift/ https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dy...

awinter-py 2y ago

they do 2 billion rides per quarter what are the 'trillions' here? ^ also that translates to ~1000 transactions per second with some assumptions; have never understood why they care so much about infra scaling 1000 tps is like 1 box