3 ms·
A quick calculation shows that all of Twitter's tweets come to about 260GB uncompressed (including 40 bytes of metadata for each tweet). The social graph would
by andr 18y ago
A quick calculation shows that all of Twitter's tweets come to about 260GB uncompressed (including 40 bytes of metadata for each tweet). The social graph would be about another 200GB. Given that the entire Twitter could fit on a high end laptop's drive, it's weird that they wouldn't have enough replication to avoid data loss. Even if you assume the data is doubled because of denormalization, it still seems fairly manageable.
- Dilpil 18y agoYou know we would love to see the calculation.
- staunch 18y agoMy guess is he's looking at a recent Twitter status_id from the public timeline[1] and multiplying it by 180 bytes. Wildly inaccurate no doubt. 1. http://twitter.com/public_timeline http://twitter.com/public_timeline
- andr 18y agoAbout a week ago there were 1.4 billion tweets and 27 million registered users, based on the tweet and user IDs. If anything, that number is slightly more than the real thing, because of deleted tweets and spam. I multiplied the tweets by 200 bytes (I added 40 bytes of metadata, and a 20 byte premium for any unicode tweets). The size of the social graph is harder to estimate. Out of a sample of 459,000 "more influential" Twitter users, the average user follows 262 other twitterers. So the information in the social graph is about 30 GB, assuming 64-bit IDs. They potentially store an edges table of the form (user1_id,user2_id) (60GB), and a denormalized list of followers and friends for each user (30GB each). This comes to about 120GB (I revise my earlier number). Yes, it's a very rough estimation, but it is in the right ballpark.
- deleted 18y ago[deleted]