3 ms·I know robots.txt doesn’t work, but is there a way to identify and block crawlers?by insane_dreamer 2y agoI know robots.txt doesn’t work, but is there a way to identify and block crawlers?