8 ms·
robots.txt is a somewhat useful tool for keeping search engines in line, because it's rather easy to prove that a search engine ignores robots.txt: when a noind
by majewsky 2y ago
robots.txt is a somewhat useful tool for keeping search engines in line, because it's rather easy to prove that a search engine ignores robots.txt: when a noindex page shows up in SERPs. This evidence trail does not exist for AI crawlers.
- ccgreg 2y agoI'd say a bigger problem is that people disagree about the meaning of nofollow and noindex.