15 ms·

Uber’s Big Data Platform: 100+ Petabytes with Minute Latency

211 points by eaguyhn 8y ago

Iwan-Zotow 8y ago
What exactly is "Big" here? It is about 1000 hard drives, several racks...
- TomVDB 8y ago
  I think you're off by a factor of 100?
  - Iwan-Zotow 8y ago
    You think wrong 100TB SSD from Nimbus, https://nimbusdata.com/products/exadrive-platform/scalable-ssds/ https://nimbusdata.com/products/exadrive-platform/scalable-s...
    - saagarjha 8y ago
      That website doesn't list a price, but I doubt that running a rack of these would be cost-effective. I wouldn't be surprised if those drives have a cost per terabyte at least a couple of times that of commodity drives.
    - samontar 8y ago
      Are you using this in prod? I’d be very surprised. I wouldn’t use this for prod data.
      - Iwan-Zotow 8y ago
        Not yet, but people are looking into them My point is, you could fit all those Uber data (I know, I know, replication, sync etc) into racks in the SINGLE well cooled room in datacenter, managed by 2 guys per shift. And this is not "Big" as far as I can see. Probably will speed up their queris/analytics as well all things being local
- randyrand 8y ago
  Ya, big data only starts at 100,000,000 PB. Everyone knows that. There is no size requirement. It's more about what you collect, the frequency, the coverage, and how you use it.
  - Iwan-Zotow 8y ago
    Of course there is And anything you could fit into, basically, one well cooled room is not "Big" anymore, sorry to tell you that
    - PhasmaFelis 8y ago
      "Big data" is anything that's too big to cram into a standard database and still access in reasonable times. There's a practical definition, it doesn't just mean "YUUUGE".
- deleted 8y ago
  [deleted]
- nemothekid 8y ago
  Usually the “big” qualifier is a fuction of RAM, not hard disk space. Getting hundreds of petabytes of data onto persistent storage in a “large” room has been possible for many years now. “Big” data was never about how to store the data.
  - Iwan-Zotow 8y ago
    What I'm trying to say, as soon as you could fit data and processing unit(s) into one well cooled room in datacenter, managed by two guys per shift, it is not "Big" problem anymore. Making it all local will probably speed-up their queries/analytics enormously as well
- teraflop 8y ago
  In my book, "bigger than 99.9% of organizations will ever encounter" is big enough to qualify as "big".
bradhe 8y ago
I've been following Uber's big data platform engineering for a while, this is a really interesting update. Specifically, it's interesting how well their Gen 3 stack held up. Also interesting choice to solve the incremental update problem at storage time instead of inserting another upstream ETL process (which would be incredibly expensive at this level of scale I'm sure). Also interesting: A lot of companies, you look at their big data ecosystem and it's littered with tons of tools. Uber seems like they've always done a good job keeping that pared down, which indicates to me that their team knows what they're doing for sure.
- iblaine 8y ago
  Being litter with tools is not necessarily a bad thing. A tool is a modular reusable component that acts a building block for something more complex. Consider Netflix. They are big open source contributors to the data industry. Many of their tools interest me because I can see their value outside of Netflix. Uber on the other hand is solving esoteric problems with esoteric solutions.
  - kerng 8y ago
    This is the tool they released: https://github.com/uber/hudi https://github.com/uber/hudi
- artwr 8y ago
  I have only heard second hand, but internally Uber has always had a large litter of tools. There is always a little bit chaos edited out from what is presented to the outside world. They run several container schedulers (YARN, Mesos, Myriad), they used to run several separate workflow orchestrators. The proliferation of tools allows to test out different ideas in production conditions which is not bad per se. But the duplication of efforts or variability has to come with some advantage to continue doing it. Infrastructure engineers wanting to reinvent the wheel can create a lot of technical debt fast, and leveraged one at that. This takes nothing away from the great work Uber has been doing in the area. They have definitely set a great direction for their platform.
saganus 8y ago
Interesting that they use the term "driver-partner" in some parts but just "driver" in others. I guess they want to avoid liability as much as possible? Would it really be possible to use a blog post in a legal proceeding to determine whether Uber has drivers or partners?
- riteshpatel 8y ago
  Pretty sure that's because in some cities the driver is dispatched by a local car company and the driver works for them, not Uber. I've seen that in a few places.
- Maro 8y ago
  In some countries, eg. here in the UAE, drivers have to employees. Most drivers are from India/Pakistan, so they need a VISA, and somebody has to sponsor it. Also, a lot of countries have laws saying [taxi] drivers _have_ to be employees, irrespective of VISA stuff.
Invictus0 8y ago
I was wondering how Uber could possibly need 100PB of space; but if you consider that they've served roughly 10 billion rides, it actually only comes out to roughly 100 kilobytes per ride.
- NeedMoreTea 8y ago
  Which is about 99.7kb more than I would hope and expect as a customer.
  - singingboyo 8y ago
    I think there are tradeoffs, for example, they probably store the route, which is good (refunds for really terrible routes) but bad (tracking). That's quite a bit of space, probably the big one. Rating and possibly comments, again could be pretty big. Multiple UUIDs or whatever (cust/driver IDs, CC txn IDs) will take up some space as well. 300B is probably a fair bit too low. That said, 100kB does seem high. 10kB or so would seem more appropriate, as a very rough ballpark.
    - rasen58 8y ago
      They probably need to store a ton of things (like maps) that aren't data though, so probably not fair to say each ride is 100kb.
      - Invictus0 8y ago
        Even if they have 20PB of maps, which is probably way too high, it's still 80kb.
    - NeedMoreTea 8y ago
      Almost none of which they need to keep. Route for 30 days perhaps, start and end point for longer. I imagine most ratings are just that with no comments.
      - 8y ago
faizshah 8y ago
Where exactly does Flink/AthenaX fall into Uber's stack?
georgewfraser 8y ago
Reading this, I can’t help but think Uber would be better off adopting one of the commercial data warehouses that separates compute from storage: Snowflake or BigQuery. They have full support for updates, they support huge scale, and because they’re more efficient the cost is comparable to Presto in spite of the margin. You can ingest huge quantities of updates if you batch them up correctly, and there are commercial tools that will do the entire ingest for you (cough Fivetran). Disclosure: am CEO of Fivetran.
- sixdimensional 8y ago
  The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well. Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings. At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive. If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.). I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).
  - jfim 8y ago
    The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.
deleted 8y ago
[deleted]
manish_gill 8y ago
Good post. The Snapshot-based approach during ingestion time was the part where I couldn't figure out why it was considered a good decision during implementation? I've experimented with Parquet data on S3 for a work POC, and the latency to fetch the data/create tables/run the Spark-SQL query (running on EMR cluster) was quite noticeable. I was advised that EMR-FS would make it run quicker, but never got around to playing with that. But I guess the creating of in-memory tables using raw data snapshots would still remain true? Or maybe I missed something. Also, I take it if 24 hrs is the latency requirements for ingestion to availability of this data, obviously this isn't the data platform that is powering the real time booking/sharing of Uber rides. I'd be curious to see what is the data pipeline that powers that for Uber.
seoanalyzer28 8y ago
amazing...
philip1209 8y ago
After their data problem exceeded a single MySQL instance - hypothetically, what would have happened if they switched to Google Cloud Spanner? Ostensibly Google has a lot more than 100 petabytes in spanner. Could you still run basic queries in it without switching to hbase?
- bradhe 8y ago
  Fairly certain the Spanner service wasn’t available then.
vazamb 8y ago
I find it interesting that one of their major pain points was data schema. After having worked at places that use plain json and places that used protobuf I can highly recommend anyone starting an even mildly complex data engineering project (complexity in data or number of stakeholders) to use something like protobuf, apache arrow or a columnar format if you need it. Having a clearly defined schema that can be shared between teams (we had a specific repo for all protobuf definitions with enforced pull requests) significantly reduces the amount of headaches down the road.
buremba 8y ago
I wonder which BI tools they use for running ad-hoc queries on their Presto cluster. The user behavioral analytics is a hassle when you use SQL and generic BI solutions don't help with that. Also, I assume that they have dashboards that use pre-aggregated tables for faster results, they probably have ETL jobs for this use-case but is the pre-aggregated data stored on HDFS as well?
tomnipotent 8y ago
Hudi sounds a lot like event sourcing but at data warehousing scale - a change log backed up by a snapshot of the latest updates.

211 points by eaguyhn 8y ago

Iwan-Zotow 8y ago

What exactly is "Big" here? It is about 1000 hard drives, several racks...

TomVDB 8y ago
I think you're off by a factor of 100?
- Iwan-Zotow 8y ago
  You think wrong 100TB SSD from Nimbus, https://nimbusdata.com/products/exadrive-platform/scalable-ssds/ https://nimbusdata.com/products/exadrive-platform/scalable-s...
  - saagarjha 8y ago
    That website doesn't list a price, but I doubt that running a rack of these would be cost-effective. I wouldn't be surprised if those drives have a cost per terabyte at least a couple of times that of commodity drives.
  - samontar 8y ago
    Are you using this in prod? I’d be very surprised. I wouldn’t use this for prod data.
    - Iwan-Zotow 8y ago
      Not yet, but people are looking into them My point is, you could fit all those Uber data (I know, I know, replication, sync etc) into racks in the SINGLE well cooled room in datacenter, managed by 2 guys per shift. And this is not "Big" as far as I can see. Probably will speed up their queris/analytics as well all things being local
randyrand 8y ago
Ya, big data only starts at 100,000,000 PB. Everyone knows that. There is no size requirement. It's more about what you collect, the frequency, the coverage, and how you use it.
- Iwan-Zotow 8y ago
  Of course there is And anything you could fit into, basically, one well cooled room is not "Big" anymore, sorry to tell you that
  - PhasmaFelis 8y ago
    "Big data" is anything that's too big to cram into a standard database and still access in reasonable times. There's a practical definition, it doesn't just mean "YUUUGE".
deleted 8y ago
[deleted]
nemothekid 8y ago
Usually the “big” qualifier is a fuction of RAM, not hard disk space. Getting hundreds of petabytes of data onto persistent storage in a “large” room has been possible for many years now. “Big” data was never about how to store the data.
- Iwan-Zotow 8y ago
  What I'm trying to say, as soon as you could fit data and processing unit(s) into one well cooled room in datacenter, managed by two guys per shift, it is not "Big" problem anymore. Making it all local will probably speed-up their queries/analytics enormously as well
teraflop 8y ago
In my book, "bigger than 99.9% of organizations will ever encounter" is big enough to qualify as "big".

bradhe 8y ago

I've been following Uber's big data platform engineering for a while, this is a really interesting update. Specifically, it's interesting how well their Gen 3 stack held up. Also interesting choice to solve the incremental update problem at storage time instead of inserting another upstream ETL process (which would be incredibly expensive at this level of scale I'm sure). Also interesting: A lot of companies, you look at their big data ecosystem and it's littered with tons of tools. Uber seems like they've always done a good job keeping that pared down, which indicates to me that their team knows what they're doing for sure.

iblaine 8y ago
Being litter with tools is not necessarily a bad thing. A tool is a modular reusable component that acts a building block for something more complex. Consider Netflix. They are big open source contributors to the data industry. Many of their tools interest me because I can see their value outside of Netflix. Uber on the other hand is solving esoteric problems with esoteric solutions.
- kerng 8y ago
  This is the tool they released: https://github.com/uber/hudi https://github.com/uber/hudi
artwr 8y ago
I have only heard second hand, but internally Uber has always had a large litter of tools. There is always a little bit chaos edited out from what is presented to the outside world. They run several container schedulers (YARN, Mesos, Myriad), they used to run several separate workflow orchestrators. The proliferation of tools allows to test out different ideas in production conditions which is not bad per se. But the duplication of efforts or variability has to come with some advantage to continue doing it. Infrastructure engineers wanting to reinvent the wheel can create a lot of technical debt fast, and leveraged one at that. This takes nothing away from the great work Uber has been doing in the area. They have definitely set a great direction for their platform.

saganus 8y ago

Interesting that they use the term "driver-partner" in some parts but just "driver" in others. I guess they want to avoid liability as much as possible? Would it really be possible to use a blog post in a legal proceeding to determine whether Uber has drivers or partners?

riteshpatel 8y ago
Pretty sure that's because in some cities the driver is dispatched by a local car company and the driver works for them, not Uber. I've seen that in a few places.
Maro 8y ago
In some countries, eg. here in the UAE, drivers have to employees. Most drivers are from India/Pakistan, so they need a VISA, and somebody has to sponsor it. Also, a lot of countries have laws saying [taxi] drivers _have_ to be employees, irrespective of VISA stuff.

Invictus0 8y ago

I was wondering how Uber could possibly need 100PB of space; but if you consider that they've served roughly 10 billion rides, it actually only comes out to roughly 100 kilobytes per ride.

NeedMoreTea 8y ago
Which is about 99.7kb more than I would hope and expect as a customer.
- singingboyo 8y ago
  I think there are tradeoffs, for example, they probably store the route, which is good (refunds for really terrible routes) but bad (tracking). That's quite a bit of space, probably the big one. Rating and possibly comments, again could be pretty big. Multiple UUIDs or whatever (cust/driver IDs, CC txn IDs) will take up some space as well. 300B is probably a fair bit too low. That said, 100kB does seem high. 10kB or so would seem more appropriate, as a very rough ballpark.
  - rasen58 8y ago
    They probably need to store a ton of things (like maps) that aren't data though, so probably not fair to say each ride is 100kb.
    - Invictus0 8y ago
      Even if they have 20PB of maps, which is probably way too high, it's still 80kb.
  - NeedMoreTea 8y ago
    Almost none of which they need to keep. Route for 30 days perhaps, start and end point for longer. I imagine most ratings are just that with no comments.
    - 8y ago

faizshah 8y ago

Where exactly does Flink/AthenaX fall into Uber's stack?

georgewfraser 8y ago

Reading this, I can’t help but think Uber would be better off adopting one of the commercial data warehouses that separates compute from storage: Snowflake or BigQuery. They have full support for updates, they support huge scale, and because they’re more efficient the cost is comparable to Presto in spite of the margin. You can ingest huge quantities of updates if you batch them up correctly, and there are commercial tools that will do the entire ingest for you (cough Fivetran). Disclosure: am CEO of Fivetran.

sixdimensional 8y ago
The pattern I have seen in many current big data architectures reflects quite well a build vs buy decision. Organizations that have the resources tend to go open source with a lot of custom tools (such as Uber, or FAANG). Those companies tend to be the "makers" of such tech as well. Organizations that don't have those resources or in-house experience, or desire to build, rely on the commercial licensing for open source offerings, cloud based or traditional commercial offerings. At these scales of data and time horizons, owning the engineering resources capable of supporting the tools might be necessary, but surely expensive. If one can go with a stable, well known commercial offering that reaches the desired scale but keeps the volumes of data in open standard formats, I think that is a good compromise. Many commercial vendors have gone that way, for example, Microsoft recently went all in on integrating Spark and HDFS closely with SQL Server 2019, and a lot of other database vendors have already done that as well (e.g. HP with Vertica on Hadoop, etc.). I also think it's possible that at these scales and performance, people doing that big data work really are on the bleeding edge and therefore, innovation and new development is almost a requirement to make it all work together and meet the aggressive performance and efficiency benchmarks desired. Especially when having to do complex things like handling both the speed and batch layers in one unifying architecture (as in lambda architecture).
- jfim 8y ago
  The problem with commercial offerings is that at larger scales, there often needs to be special optimizations for large scales, which don't apply to most customers of that vendor. For example, imagine that System X can scale well up to 1 terabyte of data, but that after that it becomes increasingly complex to handle more data and it requires special optimizations to maintain an acceptable level of performance. From System X's perspective as a vendor, 99% of their customers are perfectly fine with the performance of System X and just want more features (that the vendor can charge for upgrading to the next version of System X). On the other side, there's that annoying 1% of customers just keeps on asking for more and more performance optimizations that 99% of their customers don't care about. From the vendors' perspective, it makes more sense to invest development efforts into features (that can translate into more money) than in performance optimizations that only appeal to a narrow segment of their users. For the company that requires these optimizations, since the commercial codebase is proprietary, they're locked in and have no easy way out. That's pretty much why the large tech companies invest in owning their own part of the stack. It mitigates a scaling risk, keeps expertise in house and ensures that the solution that is developed in house matches 100% with the often specialized needs of the business.

deleted 8y ago

[deleted]

manish_gill 8y ago

Good post. The Snapshot-based approach during ingestion time was the part where I couldn't figure out why it was considered a good decision during implementation? I've experimented with Parquet data on S3 for a work POC, and the latency to fetch the data/create tables/run the Spark-SQL query (running on EMR cluster) was quite noticeable. I was advised that EMR-FS would make it run quicker, but never got around to playing with that. But I guess the creating of in-memory tables using raw data snapshots would still remain true? Or maybe I missed something. Also, I take it if 24 hrs is the latency requirements for ingestion to availability of this data, obviously this isn't the data platform that is powering the real time booking/sharing of Uber rides. I'd be curious to see what is the data pipeline that powers that for Uber.

seoanalyzer28 8y ago

amazing...

philip1209 8y ago

After their data problem exceeded a single MySQL instance - hypothetically, what would have happened if they switched to Google Cloud Spanner? Ostensibly Google has a lot more than 100 petabytes in spanner. Could you still run basic queries in it without switching to hbase?

bradhe 8y ago
Fairly certain the Spanner service wasn’t available then.

vazamb 8y ago

I find it interesting that one of their major pain points was data schema. After having worked at places that use plain json and places that used protobuf I can highly recommend anyone starting an even mildly complex data engineering project (complexity in data or number of stakeholders) to use something like protobuf, apache arrow or a columnar format if you need it. Having a clearly defined schema that can be shared between teams (we had a specific repo for all protobuf definitions with enforced pull requests) significantly reduces the amount of headaches down the road.

buremba 8y ago

I wonder which BI tools they use for running ad-hoc queries on their Presto cluster. The user behavioral analytics is a hassle when you use SQL and generic BI solutions don't help with that. Also, I assume that they have dashboards that use pre-aggregated tables for faster results, they probably have ETL jobs for this use-case but is the pre-aggregated data stored on HDFS as well?

tomnipotent 8y ago

Hudi sounds a lot like event sourcing but at data warehousing scale - a change log backed up by a snapshot of the latest updates.