4 ms·
I figure Google has released a browser simply (as with their Google toolbar) so they can spider all the 'dark web' pages everyone visits. Evidence? Look at all
by zandorg 17y ago
I figure Google has released a browser simply (as with their Google toolbar) so they can spider all the 'dark web' pages everyone visits. Evidence? Look at all those eBay search results when eBay prevents spidering.
- JeffJenkins 17y agoWhere is eBay preventing spidering? Their robots.txt doesn't block off anything significant, and while it says that robots should ask permission Google would certainly have that. I also don't remember any issues crawling it with the startup I work at except that the site is massive and extremely time-sensitive.
- zandorg 17y agoI remember a few years ago, an auction aggregator tried to spider eBay (just incrementing the auction ID) and eBay basically sued them, or blocked it off - either way, the aggregator went out of business. But that was 2001. So I guess things have changed (and thanks for informing me).