3 ms·

Having worked with the Twitter firehose, I can say with some certainty that the volume of data is measured in TBs per day. Sifting through a subset of the data

Having worked with the Twitter firehose, I can say with some certainty that the volume of data is measured in TBs per day. Sifting through a subset of the data to make correlations is working with PBs of data. Source: Former employee of one of the few companies who got the "push" firehose and not just the API.

gregsadetsky 4y ago
Agreed. Back in a distant 2011: "1 terabyte per day of data [...] - that is how much is produced by the firehose of 250 million tweets a day" [0] i.e., ~4kb per tweet. So, 2ish Tb/day of text + metadata at current rates? [0] https://techcrunch.com/2011/12/07/datasift-historical-tweets/ https://techcrunch.com/2011/12/07/datasift-historical-tweets...
- sudosysgen 4y ago
  Definitely much more. I guarantee the amount of metadata per tweet increased massively, and also now there's things like pictures and videos. More like 40-60TiB if I had to guess.
  - gregsadetsky 4y ago
    Right, I was excluding media from the equation. This other 2010 source [0] said "Twitter generates 8 terabytes of data per day". And that's 12 years ago... [0] https://techcrunch.com/2010/09/17/twitter-seeing-6-billion-api-calls-per-day-70k-per-second/ https://techcrunch.com/2010/09/17/twitter-seeing-6-billion-a...
deleted 4y ago
[deleted]