4 ms·
>The argument is that you are stealing resources (computer time) from the site owner. That's why 3taps is getting the data from google's cache without touching
by learc83 14y ago
>The argument is that you are stealing resources (computer time) from the site owner.
That's why 3taps is getting the data from google's cache without touching craigslist servers.
- smackfu 14y agoAnd makes it impossible to block them as a side effect.
- unreal37 14y agoIt's not impossible. You just tell google not to cache.
- slig 14y agoHow the heck are they scraping Google without being banned or rate limited?
- TomAnthony 14y agoThere are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies. The two that jump to mind are Authority Labs and SEOmoz. I guess: a shed load of proxies. :)
- nmridul 14y agoAmazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...