4 ms·
If I understand the details of the link, it was a particular feature of robots.txt that was considered undocumented/unsupported that Google dropped support for.
by saalweachter 3y ago
If I understand the details of the link, it was a particular feature of robots.txt that was considered undocumented/unsupported that Google dropped support for.
I think the point of it was that you could tell Google to crawl some pages (for links) but not index them?