4 ms·
I'm concerned this will lead to a number of meta-tags we will have to include in our ever-growing header tags. Can Pinterest and Facebook not just acknowledge r
by marquis 15y ago
I'm concerned this will lead to a number of meta-tags we will have to include in our ever-growing header tags. Can Pinterest and Facebook not just acknowledge robots.txt or we invent a new tag like rel=nofollow?
- guan 15y agorobots.txt is for web crawlers that follow links, it doesn't usually apply to robots that fetch a single page based on a user request. But they could invent and adopt a new tag.