2 ms·
You may be able to get away with scraping a portion of the site, but I still would find it very hard to believe you'd be able to scrape even 50%. Let's do some
by mrduncan 16y ago
You may be able to get away with scraping a portion of the site, but I still would find it very hard to believe you'd be able to scrape even 50%.
Let's do some math, these are all based on numbers from this past June which have likely only gone up since then [1]:
65 million tweets per day / 20 tweets per page = 3.25 million page views per day
Just to keep up with the stream, you'd need to do about 3.25 million page views per day or a little over 1.5 million to get half of it. Again, I'd find it very hard to believe that nobody at Twitter would catch on.
[1] http://techcrunch.com/2010/06/08/twitter-190-million-users/ http://techcrunch.com/2010/06/08/twitter-190-million-users/
- ahi 16y agoThey would catch on to the fact that they were being scraped, but they wouldn't be able to identify which 3.25 million were scrapes.