3 ms·
To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data. This is if you only collect username, timestamp a
by ohadron 8y ago
To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data.
This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos.
Full calculation here:
https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQfkbzqtCv0pvNuxU3zbMiR-iQ/edit?usp=sharing https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...
- thirit 8y agoSince a couple of years ago the tweet limit was bumped to 280 characters, so I assume the amount is double of what you calculated.
- ohadron 8y agoGood point! Fixing my calculation. So it's between 0.3gb (typical tweet is ~30 chars) and 6.8gb per second
- keithwinstein 8y agoI believe your calculations are too high by a factor of 1000... 53 bytes/(typical tweet) * 6000 typical tweets/second is 318 KB (or 0.3 MB) per second, not 318 MB.
- ohadron 8y agoYikes, you are totally right. Fixed