4 ms·
To catch and block scripts scraping Yelp in ways they don't like. If you fetch that url, you are most likely a bot that isn't following the rules and can be
by ageitgey 9y ago
To catch and block scripts scraping Yelp in ways they don't like.
If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.
- yeukhon 9y agoI see. Thanks, but then why so many?
- timr 9y agoTo make it hard for people to write code to get around it.
- yeukhon 9y agoWell, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338).
- timr 9y agoI assume that it's supposed to be regularly updated, yes.
- raverbashing 9y agoI wonder if any of their pages has a hidden link to one of those entries in robots.txt