3 ms·
I'm still not getting the futz they're making over their "scale". So your inbound load is 7000 tweets/sec or roughly 250 MBit/s (assuming 4k per tweet). Then y
by moe 15y ago
I'm still not getting the futz they're making over their "scale".
So your inbound load is 7000 tweets/sec or roughly 250 MBit/s (assuming 4k per tweet).
Then you fan that out to (assuming) 20 append-only mailboxes on average.
Perhaps my assumptions are far off, but I'm only arriving at a couple GBit/s here and a low two digit number of terabytes/storage per year.
This sounds like "a couple racks" to me, not like "a couple datacenters".
- gnuvince 15y agoAs I understand it -- and I might be wrong here -- the inbound traffic is not the real problem; it's the distribution to all the followers. If you have 7000 tweets coming in and you need to send it to the 120 followers of all these posters in real time, that makes a lot of data shuffling.
- lenn0x 15y agoHere is an old presentation from a year ago. http://www.slideshare.net/nkallen/q-con-3770885 http://www.slideshare.net/nkallen/q-con-3770885 Now, at the time they were doing peak 2000 tweet/s. The fan-out was 1.2M deliveries a second... So if we go with the current 600:1 ratio at 7000/s, that's about 4.2M/s. I actually know it's much higher now since I work there but other things to consider is we have a large data warehouse, search, API, pipelines to external parties for the firehose, logging at terabytes an hour, in-house metric collection doing 3M writes/s, etc. http://www.scribd.com/doc/59830692/Cassandra-at-Twitter http://www.scribd.com/doc/59830692/Cassandra-at-Twitter It add's up very fast.