3 ms·
Does anyone know what the current rate of tweets per second is? This would influence what it's possible to do with the streams.
by cperciva 17y ago
Does anyone know what the current rate of tweets per second is? This would influence what it's possible to do with the streams.
- ivankirigin 17y agoYou can read every tweet from a single machine. It's just a matter of getting acces to the steaming API.
- cperciva 17y agoYou can read every tweet from a single machine. Sure, but the rate of tweets will determine how much processing you can do per tweet given a certain amount of computing power.
- Tichy 17y agoIf you manage to save them somehow (or distribute them to different machines), you can distribute the processing, I suppose.
- cperciva 17y agoExactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.
- apgwoz 17y agoI have decent access to the track/follow apis. I just queue the tweet right away and have a bunch of workers take care of it. Something like rabbitmq works wonderfully here.
- ivankirigin 17y agoThe data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive http://wiki.apache.org/hadoop/Hive
- diego 17y agoRight now about 400 per second. http://www.twitpocalypse.com/ http://www.twitpocalypse.com/