4 ms·

I've wondered how much hardware it takes to store and search over every Tweet. I figure the total number of Tweets is in the low trillions. I know Twitter does

I've wondered how much hardware it takes to store and search over every Tweet. I figure the total number of Tweets is in the low trillions. I know Twitter does this themselves and presumably http://support.gnip.com/apis/search_full_archive_api/ http://support.gnip.com/apis/search_full_archive_api/ uses the same backend servers. Does Twitter have any public info on how much hardware is dedicated to this?

strictnein 9y ago
It's sometimes mentioned in their blog posts: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... Not a lot of specifics, but there's some interesting tidbits: > Hadoop: We have multiple clusters storing over 500 PB divided in four groups (real time, processing, data warehouse and cold storage). Our biggest cluster is over 10k nodes. We run 150k applications and launch 130M containers per day.