3 ms·
Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
by sharlos201068 2y ago
Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
- danielheath 2y agoAdding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.