4 ms·
Where is eBay preventing spidering? Their robots.txt doesn't block off anything significant, and while it says that robots should ask permission Google would c
by JeffJenkins 17y ago
Where is eBay preventing spidering? Their robots.txt doesn't block off anything significant, and while it says that robots should ask permission Google would certainly have that. I also don't remember any issues crawling it with the startup I work at except that the site is massive and extremely time-sensitive.
- zandorg 17y agoI remember a few years ago, an auction aggregator tried to spider eBay (just incrementing the auction ID) and eBay basically sued them, or blocked it off - either way, the aggregator went out of business. But that was 2001. So I guess things have changed (and thanks for informing me).