3 ms·
No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at
by tibbon 17y ago
No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data.
I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big.
Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie.
As a data analyst, I'm really excited.
- axod 17y agoSure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.
- Gormo 17y ago55 million tweets/day * 140 characters/tweet ~= 14.5 GB/day, presuming a 16-bit character encoding. That's only about 20 TB for four years worth of data, or about $1500 worth of storage, before compression. That doesn't seem like it would be a significant challenge for the LOC. I'm more interested in understanding why they would want to use that amount of space to archive Twitter, as opposed to anything else.
- FlemishBeeCycle 17y agoCombination of low-hanging fruit plus inflated idea of twitter's usefulness (much like other people largely unfamiliar with today's technology culture)?
- tibbon 17y agoThere's a good bit of metadata (about another 300 characters or so on average) attached to each tweet that makes it useful. Not a huge amount more, but considering the growth curve... yea 20TB is about right.