stuck_in_matrix

23 ms·

1. 1.▲
   by stuck_in_matrix 8y ago

   Python is perfectly fine for these type of tasks. I ingest all of Reddit in real-time ( https://pushshift.io ) and also ingest Gab.com and several others (Stackoverflow, etc.) and at most one or two CPU cores are at 10-15%. Also,
2. 2.▲
   by stuck_in_matrix 8y ago

   When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution. I'm rewriting th
3. 3.▲
   by stuck_in_matrix 8y ago

   It's variable based on what's going on at the time but I've seen upwards of 7k tweets a second for the sections of the timeline that I've ingested using this technique. Someone suggested trying it when the New Year start
4. 4.▲
   by stuck_in_matrix 8y ago

   This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
5. 5.▲
   by stuck_in_matrix 8y ago

   I am the author of this document. If anyone has any questions, I'd be happy to answer them!