43 ms·
You can read every tweet from a single machine. It's just a matter of getting acces to the steaming API.
by ivankirigin 17y ago
You can read every tweet from a single machine. It's just a matter of getting acces to the steaming API.
- cperciva 17y agoYou can read every tweet from a single machine. Sure, but the rate of tweets will determine how much processing you can do per tweet given a certain amount of computing power.
- Tichy 17y agoIf you manage to save them somehow (or distribute them to different machines), you can distribute the processing, I suppose.
- cperciva 17y agoExactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.
- apgwoz 17y agoI have decent access to the track/follow apis. I just queue the tweet right away and have a bunch of workers take care of it. Something like rabbitmq works wonderfully here.
- ivankirigin 17y agoThe data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive http://wiki.apache.org/hadoop/Hive