3 ms·

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any pr

by amluto 2y ago

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly expensive. 1.7PB of Parquet files in on-prem or collocated object storage, even replicated a zillion times, is quite cheap. And quite a few companies and open-source projects are currently competing aggressively to provide awesome tools for querying that data. The hot data would fit on basically anything — the choice should be about robustness and barely even consider cost per TiB. Datomic got written up recently and seems credible for this type of application. FoundationDB is bulletproof. Postgres could probably handle it without breaking a sweat, although active/active replication isn’t free. Heck, writes straight to a warehouse with a cache in front to help with reads seems credible — Uber rides rarely go for longer than a couple hours, and back-of-the-envelope math suggests that the total data rate is maybe 50GB/hour. An entire day of data for an entire country would fit on a single very ordinary commodity server, and the live data for the entire world would fit on one mildly beefy server. The indexes involved sound straightforward.

crmd 2y ago
The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.
- 8ytecoder 2y ago
  Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.
  - nikolajan 2y ago
    > It’s done even in banking What are you basing this on, banks have less real time constraints when it comes to certain data but thats rare. You will almost never see a bank, processor, etc. archive their system of record data.
- jiggawatts 2y ago
  Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.
- amluto 2y ago
  > This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? - Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient. - Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data? - Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record. So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.
  - nikolajan 2y ago
    >This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)
    - kikimora 2y ago
      Capture can happen days after successful authorization.
- lozenge 2y ago
  "then developers would have multiple systems and namespaces to deal with in order to query on transactions." Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.
smileysteve 2y ago
This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all
marcyb5st 2y ago
In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).

by amluto 2y ago

crmd 2y ago

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

8ytecoder 2y ago
Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.
- nikolajan 2y ago
  > It’s done even in banking What are you basing this on, banks have less real time constraints when it comes to certain data but thats rare. You will almost never see a bank, processor, etc. archive their system of record data.
jiggawatts 2y ago
Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.
amluto 2y ago
> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? - Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient. - Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data? - Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record. So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.
- nikolajan 2y ago
  >This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)
  - kikimora 2y ago
    Capture can happen days after successful authorization.
lozenge 2y ago
"then developers would have multiple systems and namespaces to deal with in order to query on transactions." Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.

smileysteve 2y ago

This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all

marcyb5st 2y ago

In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).