5 ms·

> None of their application is data heavy Really?? They have O(thousands) of drivers in O(hundreds) of cities with the app open, sending and receiving data ap

by tln 7y ago

> None of their application is data heavy Really?? They have O(thousands) of drivers in O(hundreds) of cities with the app open, sending and receiving data approximately all day. What companies in your mind are data heavy?

disordinary 7y ago
Not to mention the hundreds of thousands of passengers waiting for and riding in vehicles. The "data heavy" businesses often have the ability to cache their data in CDNs. Uber's data is realtime and dynamic, you scale that the hard way.
drblast 7y ago
While not a "tiny amount of data" Uber's problem seems like one of those embarrassingly parallel ones where partitioning by location is trivial, and accuracy of all the data isn't as important as knowing everyone's location right now. I'd guess any relatively popular MMO deals with similar things under much tighter latency constraints. Data heavy to me means terabytes of data with potentially multiple dimensions, like Google or Facebook. Not to say that Uber's problem isn't challenging, but I don't expect that the amount of data is necessarily the problem.
theincredulousk 7y ago
Can't speak for OP, but I'd imagine most of the data in the system is metadata - i.e. very tiny. Sure they're moving a lot of packets, but an entire ride's data could probably fit in 100KB. If they have a million drivers active, what data do they actually need on them? Profile, GPS location, type? Few KB. Keeping the log of all rides probably takes up the most, but still, relatively small. In any case they essentially deal with small, highly compressible metadata. Think of that vs. a company like Instragram where people are uploading probably terabytes of media content every day, and they have to maintain/host all that content to be served on-demand basically forever. This is probably a low-key reason for pushing "stories" - they get a reprieve by only having to store that for 24 hours. In any case, you're looking at orders of magnitude larger data usage in all aspects.
- spookthesunset 7y ago
  Uber eats, for example, gives a guess of when your order arrives based on past data. Uber gives estimates of when your ride will show up. They do lots of stuff in fairly real time with that data and a lot of it is probably compute intense. Not at all comparable to Instagram.
- tln 7y ago
  Or YouTube, with 1000s of hours of video uploaded per minute. FWIW an Instagram pic, 1080x1080 is around 100kb. Uber has to do a lot more processing on it's data. I'm sure there are real challenges and the OSS projects from Uber I've seen/remembered on HN seem to be the type one builds because existing tools don't solve the problem well enough for their cases.
- erghoe 7y ago
  FWIW, Instagram stores Stories, privately to the user, in its "Archive" feature indefinitely. https://instagram-press.com/blog/2017/12/05/introducing-stories-highlights-and-stories-archive/ https://instagram-press.com/blog/2017/12/05/introducing-stor...
Traubenfuchs 7y ago
83 countries, 858 cities, 75 million riders, 3 million drivers. 5 billion rides in 2017. That's 14 million a day on average. I would guesstimate peak days have at least double the average. That is data heavy to me. (-; https://muchneeded.com/uber-statistics/ https://muchneeded.com/uber-statistics/
- theincredulousk 7y ago
  In the grand scheme of things it just isn't. 14 million rides a day, giving them a generous 1MB of data per ride, is "only" 14TB, 420TB/month. The data relevant to a ride for the clients like GPS location and ETA can fit into a few bytes... A single video doorbell can use 200GB / month. With only 10,000 users that is 2 Petabytes of data traffic per month. Not to mention recordings that have to be stored and hosted for streaming later on. Everything is relative (-;

by tln 7y ago

disordinary 7y ago

Not to mention the hundreds of thousands of passengers waiting for and riding in vehicles. The "data heavy" businesses often have the ability to cache their data in CDNs. Uber's data is realtime and dynamic, you scale that the hard way.

drblast 7y ago

While not a "tiny amount of data" Uber's problem seems like one of those embarrassingly parallel ones where partitioning by location is trivial, and accuracy of all the data isn't as important as knowing everyone's location right now. I'd guess any relatively popular MMO deals with similar things under much tighter latency constraints. Data heavy to me means terabytes of data with potentially multiple dimensions, like Google or Facebook. Not to say that Uber's problem isn't challenging, but I don't expect that the amount of data is necessarily the problem.

theincredulousk 7y ago

Can't speak for OP, but I'd imagine most of the data in the system is metadata - i.e. very tiny. Sure they're moving a lot of packets, but an entire ride's data could probably fit in 100KB. If they have a million drivers active, what data do they actually need on them? Profile, GPS location, type? Few KB. Keeping the log of all rides probably takes up the most, but still, relatively small. In any case they essentially deal with small, highly compressible metadata. Think of that vs. a company like Instragram where people are uploading probably terabytes of media content every day, and they have to maintain/host all that content to be served on-demand basically forever. This is probably a low-key reason for pushing "stories" - they get a reprieve by only having to store that for 24 hours. In any case, you're looking at orders of magnitude larger data usage in all aspects.

spookthesunset 7y ago
Uber eats, for example, gives a guess of when your order arrives based on past data. Uber gives estimates of when your ride will show up. They do lots of stuff in fairly real time with that data and a lot of it is probably compute intense. Not at all comparable to Instagram.
tln 7y ago
Or YouTube, with 1000s of hours of video uploaded per minute. FWIW an Instagram pic, 1080x1080 is around 100kb. Uber has to do a lot more processing on it's data. I'm sure there are real challenges and the OSS projects from Uber I've seen/remembered on HN seem to be the type one builds because existing tools don't solve the problem well enough for their cases.
erghoe 7y ago
FWIW, Instagram stores Stories, privately to the user, in its "Archive" feature indefinitely. https://instagram-press.com/blog/2017/12/05/introducing-stories-highlights-and-stories-archive/ https://instagram-press.com/blog/2017/12/05/introducing-stor...

Traubenfuchs 7y ago

83 countries, 858 cities, 75 million riders, 3 million drivers. 5 billion rides in 2017. That's 14 million a day on average. I would guesstimate peak days have at least double the average. That is data heavy to me. (-; https://muchneeded.com/uber-statistics/ https://muchneeded.com/uber-statistics/

theincredulousk 7y ago
In the grand scheme of things it just isn't. 14 million rides a day, giving them a generous 1MB of data per ride, is "only" 14TB, 420TB/month. The data relevant to a ride for the clients like GPS location and ETA can fit into a few bytes... A single video doorbell can use 200GB / month. With only 10,000 users that is 2 Petabytes of data traffic per month. Not to mention recordings that have to be stored and hosted for streaming later on. Everything is relative (-;