4 ms·
The Chilling Effects robots.txt file [1]. Does anyone know what the user-agent "Google-Legal-Removals" is for? Returning to the subject at hand, what are they
by gizmo686 12y ago
The Chilling Effects robots.txt file [1]. Does anyone know what the user-agent "Google-Legal-Removals" is for?
Returning to the subject at hand, what are they trying to do by making themselves non-index-able? If I search something on Google and a link has been removed, I still get linked to the DMCA request on chillingeffects that still has the removed url on it. If I go to the chillingeffects website, they still offer a search feature.
[1] https://www.chillingeffects.org/robots.txt https://www.chillingeffects.org/robots.txt