5 ms·
Ever wondered how much data web giants generate? Grab says 1.7MB/user/day
- hidden-spyder 5y agoSo that is 3400 Terrabytes of data in a year for 2 billion users. I used the number 2 billion because it's approximately the number of people on Facebook. Not vetted, just for a quick calculation.
- ksec 5y agoIt is 3400 Terrabytes of Data per day for 2 Billion users. Or 1241 Petabyte, 1.241 Exabyte per year. No wonder why HDD maker are happy.
- hidden-spyder 5y agoOh, yeah! You're right. In between trying to convert from MBs to TBs, I forgot that I was looking at the day scale, not year. 1.2 Exabytes a year is a lot.
- deleted 5y ago[deleted]
- sudhirj 5y agoIt’s 12 AWS Snowmobiles, which are container trucks full of hard drives. A corporation that generates that much data probably has more container trucks of furniture.
- buran77 5y agoI don't think they factor in compression here. It's likely the 1.7MB they're talking about is raw data. Google's location history for example can be compressed down to very manageable single digit percentages even with fast compression. Edit. I'm talking about Grab, the "transportation, e-commerce, payments, and financial services" company in the article, which sounds like it collects little if any multimedia from users. Also thinking more of data the company generates from the user (usage history, metadata and collection), not what the user generates and uploads. The 1.7MB figure can't really be extrapolated for a Facebook-like data set (or worse, YouTube). Social networks have a lot of mostly multimedia content created and uploaded by the user, and lower volume of company generated content like location tracking and lists of interactions.
- GordonS 5y agoIsn't it also likely that a lot of that "raw" data is comprised of JPEGs (which are already compressed)?
- pwdisswordfish0 5y agoIf it’s raw data and not simply actual disk use divided by users, I would guess the savings from compression likely pale in comparison to metadata such as indices, backups and other overhead.
- yosito 5y agoIs there any reason to believe that Facebook is collecting the same amount of data as Grab?
- boomboomsubban 5y agoIt's reasonable to assume they're collecting a comparable amount. Even if Facebook collected half as much, 1700 terabytes per day is still a lot.
- kaetemi 5y agoSo, how much of this data is speculative hoarding, and how much energy is being wasted here?
- ariejan 5y agoNow, imagine storing that on 3.5" floppy disks...
- liotier 5y agoWith some basic compression, this daily 1.7 MB will fit on a 1.44 MB 3.5" floppy... Our Father, who art in The Cloud, give us this day our daily floppy ! Don't forget to swap your floppy every morning. https://i.imgur.com/ntrjkV7.jpg https://i.imgur.com/ntrjkV7.jpg !
- eric4smith 5y agoI don’t doubt this one bit. Here in Thailand I use grab at least once every day. Sometimes up to 4 times a day. Taxi to office, from office, meals, groceries, courier service. Grab would not work without the army of motor bikes. Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data. Transactions are probably the smallest amounts. Another big one is probably all the inventory records from supermarkets, convenience stores and restaurants in probably the number one foodie city in the world - Bangkok. Then there is taking photos to prove delivery pickups and drop offs. Damn… Then travel to the Philippines, Vietnam, Indonesia - grab also just works. It’s actually quite an impressive operation and they’ve improved the app quite a bit in the last year alone. Now with the lockdowns at the moment they are super busy more than usual - they probably up that amount of daily data by amount 50%
- monsieurbanana 5y agoSeeing how prevalent grab & foodpanda has become here in Singapore, I'd say much more than 50%.
- tpxl 5y ago> Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data. Logging gps once per second every second of the day using two doubles will get you 691Kib of data. Unless you have two deliveries in flight 24/7 the math does not check out.
- chrismorgan 5y agoI believe precise GPS coordinates is 12 bytes per sample, so that’s ~1MiB/day. However, it compresses extremely well even with lossless content-unaware compression techniques (though I don’t have any real numbers), and if you’re serious about compression you can do even better still, with http://cs.joensuu.fi/~mchen/GPSTrajComp.htm http://cs.joensuu.fi/~mchen/GPSTrajComp.htm showing a technique that gets you down to ~3–6KiB per day if you’re happy with up to 10m of error.
- 5y ago
- askmike 5y ago> All that data is clearly valuable: Grab has also announced record profits. What do they say about correlation and causation?
- bryanrasmussen 5y agothat there often is one? Obviously you can't say clearly all the data is valuable, but you can say clearly some of the data is valuable because without any data they would not have any business.
- pdkl95 5y agoA useful reference for magnitude of "1MB" is that a typical (english) novel contains around 0.5 to 1 MB of ASCII text. Neuromancer is just under 500kB; Snow Crash is ~900kB. So 1.7MB/user/day is roughly equivalent to 2-3 novels written about each user every day.
- thefrenchsavage 5y agoAnybody here did DD on SPAC Altimeter Growth Corp. (AGCUU)?
- anotheryou 5y agoNumber of values would be much more interesting. An image is big, a bloated json somewhat big, and compressed or not compressed? (actually the compressed size would be more telling)