4 ms·
Most(not all) search engine crawlers seem to obey robots.txt's "Disallow: /" in my experience. But beyond that, maybe the preview mode of the site should be don
by smtddr 13y ago
Most(not all) search engine crawlers seem to obey robots.txt's "Disallow: /" in my experience. But beyond that, maybe the preview mode of the site should be done in a way to make crawlers/webscrapers return worthless results. Like make the no-sign-up site not work without javascript or constantly request captchas with annoying frequency. For a dating site, I'd assume no-sign-up mode shouldn't show any detail about the signed-up people beyond city/state, age & profile-picture.
- darkstar999 13y ago> constantly request captchas with annoying frequency That's an anti-pattern that would annoy me more than the few minutes it would take to create an account.