Not sure it applies to twitter specifically, but if you are scraping websites you should check robots.txt. The intended purpose is to direct you to where the co

Not sure it applies to twitter specifically, but if you are scraping websites you should check robots.txt. The intended purpose is to direct you to where the content actually is, so you're not wasting resources spidering through contentless pages, just because they're linked from somewhere.