3 ms·I don't know if it's illegal, but Google would follow the site's robots.txt.by debacle 14y agoI don't know if it's illegal, but Google would follow the site's robots.txt.