4 ms·
Writing a scraper is easy. So I assume the special sauce in these tools is caching, not getting banned or rated limited. Mostly could be done with Tor but still
by dplgk 7y ago
Writing a scraper is easy. So I assume the special sauce in these tools is caching, not getting banned or rated limited. Mostly could be done with Tor but still doesn't work with some sites like Craigslist. Is that right?
- omneity 7y agoIt depends on your usage. If you're parsimoniously checking for data, the website will not be hostile. If your usage requires higher frequency against a given website, you'll indeed need a proxy (which we'll be rolling relatively soon, when there is demand for it).