3 ms·
We develop and host a bunch of extranets, which without login consist of your typical authentication page. We put a robots.txt file there, and the only sites th
by lloeki 14y ago
We develop and host a bunch of extranets, which without login consist of your typical authentication page. We put a robots.txt file there, and the only sites that link there are our customers companies home sites.
Google still indexes them. The definition of "relevant" here defies my wildest imagination.
- kevinpet 14y agorobots.txt is not about indexing. It's about crawling.