5 ms·
I was wondering how Uber could possibly need 100PB of space; but if you consider that they've served roughly 10 billion rides, it actually only comes out to rou
by Invictus0 8y ago
I was wondering how Uber could possibly need 100PB of space; but if you consider that they've served roughly 10 billion rides, it actually only comes out to roughly 100 kilobytes per ride.
- NeedMoreTea 8y agoWhich is about 99.7kb more than I would hope and expect as a customer.
- singingboyo 8y agoI think there are tradeoffs, for example, they probably store the route, which is good (refunds for really terrible routes) but bad (tracking). That's quite a bit of space, probably the big one. Rating and possibly comments, again could be pretty big. Multiple UUIDs or whatever (cust/driver IDs, CC txn IDs) will take up some space as well. 300B is probably a fair bit too low. That said, 100kB does seem high. 10kB or so would seem more appropriate, as a very rough ballpark.
- rasen58 8y agoThey probably need to store a ton of things (like maps) that aren't data though, so probably not fair to say each ride is 100kb.
- Invictus0 8y agoEven if they have 20PB of maps, which is probably way too high, it's still 80kb.
- NeedMoreTea 8y agoAlmost none of which they need to keep. Route for 30 days perhaps, start and end point for longer. I imagine most ratings are just that with no comments.
- btown 8y agoBeyond the route itself, consider also: time spent in traffic (speed per geo-coordinate) for use in predicting surges, deviations between planned and taken routes, etc. The degree to which Uber has a pulse on urban traffic is fascinating and probably exceeds even city planners with access to ubiquitous traffic cameras.
- Jedd 8y agoThat's a fairly naive expectation. I had a friend head off in 2011 to work with Hailo in London, and designed much of the data ingest & storage architecture there. That was built on Cassandra, and some of the numbers were staggering, even for a (initially) single-city, < 2,000 driver system. You're not just tracking start and end point of successful rides -- you're tracking users when their app is open (both drivers and users move around before and after initiating a ride, and you need to share that data between parties), you're also tracking empty vehicles so you can identify over / under serviced areas ... doubtless lots of other real time stuff you want to know. But the big thing is you want to then have that data available for mining, improving your service, regulatory obligations, and so on.
- NeedMoreTea 8y agoOnly if you believe in retaining all data in perpetuity. I suppose it demonstrates the difference in US and European perspectives on data. Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond. Once the data has outlived that direct, primary, purpose you dispose of it or it's a famous, excessive, data breach waiting to happen. There is absolutely nothing wrong with only keeping billing data beyond say six months max.
- arbie 8y ago> Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond. The really invasive pattern-matches happen best retroactively at that scale. Age of the data does not typically impact many such analyses.
- samontar 8y agoIn California, you soon won’t be able to keep 6 months max. You have to hold a year.
- Jedd 8y ago
- pso 8y agoI think you miscalculated. I get 10MB per trip on average, assuming 10bn total trips. Maybe deduct something for maps and inefficient file formats & data structures? 1 billion bytes = 1GB, 1 billion KB = 1TB, 1 billion MB = 1PB, 10 billion * 10MB = 100PB.
- Invictus0 8y agoYes, you're right. My calculation was 2^50 bytes in a PB / 10^10 trips, which should have been multiplied by 100 to get 100PB. Which again leaves me wondering why Uber needs 100PB.
- ddalex 8y agoTinfoil hat: data that they slurp outside trips, like tracking user locations, etc
- efrank02 8y agoI do forecasting at Uber, we don’t collect data outside of trips.
- cr3ative 8y agoHas this kind of thing stopped, then? https://techcrunch.com/2016/11/28/uber-background-location-data-collection/ https://techcrunch.com/2016/11/28/uber-background-location-d...
- compcoffee 8y ago>we don’t collect data outside of trips. And it's not like Big Tech has ever misled people on this subject.
- kk58 8y agoWhat kind of problems do you work on?
- buremba 8y ago
- jandrewrogers 8y agoThe data sizing heuristic for basic vehicle positioning telemetry is a GB per vehicle per year assuming a professional driver (e.g. taxi, trucking, etc). A petabyte is about a million vehicle-years of positioning data, which isn't that much. Remember, Uber also stores application positioning telemetry as well, so it isn't just the number of vehicles.
- bloomer 8y agoIt makes me wonder if those yellow cabs are just stuffed full of hard drives to store all their historic data too. None of that data is useful or necessary. I would love to see any “insight” that would beat the knowledge of a basically competent cabbie on where and when people need rides. Uber just has lots of investor funny money that they want to spend to make it look like technology has something to do with their business when they are actually just a marketing and booking engine for independent taxi operators. Investors just get more excited and start throwing “internet money” your way if you brag about things like amount of data and engineers.
- samontar 8y agoWell, I’d expect there to be aggregate result stores as well for other dashboards that may store the same data sorted and aggregated a different way.